Image processor, image processing method and program

ABSTRACT

An image processor performing super-resolution of converting an input image with first resolution to an output image with second resolution higher than the first resolution on consecutive input images includes a prediction unit predicting the output image with the second resolution of a current frame using the input image of the current frame and the output image obtained by the super-resolution on an input image of a previous frame; a generation unit generating a reduced image with the first resolution composed of pixels at different phases of the prediction image using a prediction image obtained by the prediction of the prediction unit; a difference calculation unit calculating a difference between the input image of the current frame and the reduced image; and an addition unit adding the difference up-sampled to the second resolution to the prediction image, thus generating the output image with the second resolution of the current frame.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processor, an image processingmethod, and a program. More particularly, the present invention relatesto an image processor, an image processing method, and a program, whichare suitable for use when converting an input image to an image with ahigher resolution.

2. Description of the Related Art

In the related art, a method called super-resolution is known as amethod of obtaining an image with a higher resolution from an inputimage (for example, see Japanese Unexamined Patent ApplicationPublication No. 2008-140012)

For example, according to a technique called super-resolutionback-projection, a consecutive frame image such as a moving image, whichis input to an image processor 11, is converted to an image with ahigher resolution and output, as illustrated in FIG. 1. In the followingdescription, the image with a lower resolution being input to the imageprocessor 11 will be referred to as an LR (low resolution) image, andthe image with a higher resolution being output from the image processor11 will be referred to as an SR (super resolution) image.

The input LR image is up-sampled to an image with the same resolution asthe SR image by an up-sampler 21, and the up-sampled image is suppliedto a motion vector detector 22, a mask generator 23, and a mixer 24. Theimage processor 11 includes a buffer 25 storing an SR image which isobtained from an LR image of a previous frame immediately before thecurrent frame being processed at the current point of time. The SR imagesupplied to the buffer 25 is stored only for one frame period and issupplied to the motion vector detector 22 and a motion compensator 26.

The motion vector detector 22 detects a motion vector from the LR imagesupplied from the up-sampler 21 and the SR image supplied from thebuffer 25. The motion compensator 26 performs motion compensation usingthe motion vector supplied from the motion vector detector 22 and the SRimage supplied from the buffer 25. Specifically, the motion compensator26 predicts an SR image obtained from the LR image of the current frameusing the SR image of the previous frame immediately before the currentframe being processed and supplies the image obtained thus to the maskgenerator 23 and the mixer 24 as a prediction image.

The mask generator 23 generates a motion mask using the LR imagesupplied from the up-sampler 21 and the prediction image supplied fromthe motion compensator 26. The motion mask is information used forspecifying a region of the LR image in which a moving subject isdisplayed, and which is generated by calculating a difference betweenthe LR image and the prediction image.

The mixer 24 combines the prediction image supplied from the motioncompensator 26 and the LR image supplied from the up-sampler 21 usingthe motion mask supplied from the mask generator 23 and supplies acombined image obtained thus to a down-sampler 27 and an adder 28.Specifically, the LR image and the prediction image are subjected toweighted addition with weighting factors determined by the motion mask,thus obtaining the combined image. When generating the combined image,the weighting factors are determined such that the contribution ratio ofthe LR image becomes larger in the region where a motion occurs, wherebyimage quality deterioration of the SR image resulting from predictionerrors occurring in the region with a motion is suppressed.

The combined image obtained thus is down-sampled by the down-sampler 27,and a reduced image obtained thus is supplied to a subtractor 29. Thereduced image is an image with the same resolution as the LR image.

The subtractor 29 generates a differential image by calculating adifference between the LR image supplied to the image processor 11 andthe reduced image. The up-sampler 30 up-samples the differential imageto an image with the same resolution as the SR image, thus obtaining anenlarged image. Subsequently, an adder 28 adds the enlarged image andthe combined image, outputs an image obtained thus as an SR image of thecurrent frame, and supplies the SR image to the buffer 25 to be storedtherein.

In the image processor 11, as illustrated in FIG. 2, a combined imageP11 which is obtained by predicting the SR image of the current frame isdown-sampled to obtain a reduced image P12, and a difference between thereduced image P12 and an LR image P13 is calculated to obtain adifferential image P14. The differential image P14 is an image which isindicative of an error in the reduced image P12 used as the LR image ofthe current frame obtained by prediction with respect to the LR imageP13 of the current frame. That is to say, the differential image P14 canbe said to be an error in the combined image P11 used as the predictedSR image with respect to a correct SR image of the current frame whichshould have been obtained if there was no error.

Therefore, by up-sampling the reduced image P14 and adding a signalobtained thus to the combined image P11, an image which is furthersimilar to the correct SR image which should be obtained. That is, theobtained SR image will become an image in which the LR image is morefaithfully enlarged without any image quality deterioration.

As described above, according to the back-projection, an enlarged imagewhich is indicative of the calculated error is added to the combinedimage obtained by prediction, thus obtaining the SR image.

SUMMARY OF THE INVENTION

However, when a reduced image is generated from a combined image usingthe back-projection, pixels at a predetermined phase (position) of thecombined image are sampled, and an image composed of the sampled pixelsis obtained as the reduced image.

According to sampling theorem, a signal at the Nyquist frequency is notproperly preserved in terms of its amplitude and phase. Therefore,depending on the phase of the pixels of the combined image beingsampled, there may be a case where the waveform of the combined image isnot properly preserved at the time of generating the reduced image.

For example, it will be assumed that the waveform of the combined image,that is, the change in the pixel values of the pixels of the combinedimage in a predetermined direction, has a shape as illustrated in FIG.3. In FIG. 3, the vertical direction represents the pixel values ofpixels of an image, and the horizontal direction represents apredetermined direction of the image. In addition, each circlerepresents each pixel on the image.

Referring to FIG. 3, a waveform indicated by an arrow A11 represents thewaveform of a combined image. That is, a curve that connects adjacentpixels (i.e., the pixel values thereof) of the combined image forms thewaveform of the combined image.

In this figure, when a value between the maximum value and the minimumvalue of the pixel value of the combined image is referred to as anintermediate value, the combined image is composed of pixels which arerepeatedly arranged in the horizontal direction in the figure in theorder of a pixel having the intermediate value, a pixel having themaximum value, a pixel having the intermediate value, and a pixel havingthe minimum value. That is to say, in the example of FIG. 3, thewaveform of the combined image has a sinusoidal shape.

Now, the pixels of the combined image are sampled every other pixel inthe horizontal direction in the figure so as to generate a reducedimage.

For example, when the pixels of the combined image are sampled everyother pixel in the horizontal direction so as to generate a reducedimage by starting with the second pixel from the left in the figure, thepixels of the combined image indicated by an arrow A12 are sampled. Thepixels of the combined image indicated by the arrow A12 have pixelvalues of either the maximum value or the minimum value, and thewaveform of the reduced image will have the same sinusoidal shape as thewaveform of the combined image.

On the contrary, for example, when the pixels of the combined imageindicated by the arrow A11 are sampled every other pixel in thehorizontal direction so as to generate a reduced image by starting withthe first pixel from the left in the figure, the pixels of the combinedimage indicated by an arrow A13 are sampled. The pixels of the combinedimage indicated by the arrow A13 have pixel values of the intermediatevalue, and the waveform of the reduced image will have a flat shapewithout amplitude differently from the waveform of the combined image.

As described above, when the pixels of the combined image are sampled atthe Nyquist frequency of the combined image (i.e., the half-frequency ofthe combined image) so as to generate the reduced image, depending on asampling position, there may be a case where the original waveform ofthe combined image is not properly preserved in the obtained reducedimage.

When the waveform of the combined image obtained by prediction is notproperly preserved in the reduced image, the error between the combinedimage and the LR image is not properly detected even when the differencebetween the reduced image and the LR image is calculated. As a result,errors which were not detected are accumulated in the SR image.

For example, as illustrated in FIG. 4, when the waveform of the combinedimage is properly preserved at the time of generating the reduced image,errors in the combined image are corrected, whereby an SR image can beobtained in which the waveform of the LR image is properly preserved. InFIG. 4, the vertical direction represents the pixel values of pixels ofan image, and the horizontal direction represents a predetermineddirection of the image. Moreover, arrows A21 to A27 represent a combinedimage, the pixels of the combined image being sampled, a reduced image,an LR image, a differential image, an enlarged image, and an SR image,respectively. In addition, each circle represents each pixel on theimage.

It will be assumed that a combined image as indicated by the arrow A21is obtained having the same waveform as the waveform illustrated in FIG.3. Moreover, it will be assumed that among the pixels of the combinedimage, pixels as indicated by the arrow A22, having pixel values of themaximum value and the minimum value are sampled, whereby a reduced imageis generated.

In this case, a reduced image as indicated by the arrow A23 is obtainedhaving the same waveform as the original waveform of the combined image,in which the sinusoidal waveform of the combined image is preserved.Moreover, the waveform of the LR image has a flat shape without anychange in the horizontal direction of the figure, as indicated by thearrow A24. The pixels of the LR image have pixel values of anintermediate value between the maximum value and the minimum value ofthe pixel values of the pixels of the reduced image.

When a difference between the reduced image and the LR image iscalculated by the subtractor 29, a differential image having a waveformas indicated by the arrow A25 is obtained. The waveform of thedifferential image has a shape such that the waveform of the reducedimage is reversed in the vertical direction of the figure with respectto the position of the intermediate value.

When the differential image is up-sampled, an enlarged image asindicated by the arrow A26 is obtained in which the original waveform ofthe differential image is preserved. When the enlarged image is added tothe combined image indicated by the arrow A21 by the adder 28, theerrors generated at the time of generating the combined image arecorrected, whereby an SR image as indicated by the arrow A27 can beobtained having the same waveform as the waveform of the LR image.

As described above, according to the back-projection, when the originalwaveform of the combined image is properly preserved at the time ofgenerating the reduced image, it is possible to increase the resolutionof the LR image without any image quality deterioration.

On the contrary, for example, as illustrated in FIG. 5, when thewaveform of the combined image is not properly preserved at the time ofgenerating the reduced image, the errors in the combined image are notcorrected, whereby errors are accumulated in the obtained SR image at acertain phase. In FIG. 5, the vertical direction represents the pixelvalues of pixels of an image, and the horizontal direction represents apredetermined direction of the image. Moreover, arrows A31 to A37represent a combined image, the pixels of the combined image beingsampled, a reduced image, an LR image, a differential image, an enlargedimage, and an SR image, respectively. In addition, each circle in thefigure represents each pixel on the image.

It will be assumed that a combined image as indicated by the arrow A31is obtained having the same waveform as the waveform illustrated in FIG.3. Moreover, it will be assumed that among the pixels of the combinedimage, pixels as indicated by the arrow A32, having pixel values of themaximum value and the minimum value are sampled, whereby a reduced imageis generated.

In this case, a reduced image as indicated by the arrow A33 is obtainedhaving a flat shape without any change in pixel values, in which thesinusoidal waveform of the combined image is not preserved. Moreover,the waveform of the LR image has a flat shape without any change in thehorizontal direction of the figure, as indicated by the arrow A34. Thepixels of the LR image have pixel values of an intermediate value.

When a difference between the reduced image and the LR image iscalculated by the subtractor 29, a differential image having a waveformas indicated by the arrow A35 is obtained. The waveform of thedifferential image has a shape such that it is flat in the horizontaldirection in the figure, and the pixel values of the pixels thereof havethe same value. In an ideal case, at the phase of the pixels of thecombined image having the maximum value and the minimum value, adifference between the maximum value or minimum value and theintermediate value should be detected as an error. However, in theexample of FIG. 5, since the sampling position at the time of generatingthe reduced image is not proper, the above-mentioned error is notdetected.

When the differential image is up-sampled, an enlarged image asindicated by the arrow A36 is obtained in which the original waveform ofthe differential image is preserved. When the enlarged image is added tothe combined image indicated by the arrow A31 by the adder 28, theerrors generated at the time of generating the combined image are notcorrected, whereby an SR image as indicated by the arrow A37 can beobtained having a different waveform from that of the LR image. That isto say, in the SR image, the errors at the certain phase generated atthe time of generating the combined image remain unremoved.

Furthermore, the obtained SR image is used for the prediction of an SRimage of a subsequent frame as it is. As a result, at a certain phase ofa certain frequency where errors are irremovable, errors generated ineach frame, namely noise components generated by the prediction areaccumulated in the SR image. According to the back-projection, sincepixels at a predetermined phase of the combined image of each frame aresampled at the time of generating the reduced image, there is a concernthat errors are accumulated at positions (phases) of the pixels of thecombined image, which are not sampled.

When errors generated in each frame are accumulated in the SR image, theimage quality of the SR image of each frame deteriorates. For example,in the example of FIG. 5, a straight line-shaped noise which is long ina direction perpendicular to the horizontal direction in the figure,namely, a comb-like noise appears in the SR image.

Furthermore, in the field of super-resolution, a technique called a Mapmethod is known as a method of suppressing accumulation of noise in anSR image. The Map method utilizes characteristics of images having astrong spatial correlation to apply feedback to an image obtained bypredicting an SR image with spatial constraints placed on the errorsrelative to an LR image. Therefore, accumulation of noise is prevented.However, although the Map method is able to suppress the accumulation ofnoise, since the edge portions are lost, the SR image will becomeblurred, resulting in image quality deterioration.

As described above, in super-resolution, when an SR image is obtainedfrom an LR image, it is difficult to suppress image qualitydeterioration of the SR image.

It is therefore desirable to improve further the image quality of imageswhen an input image is converted to an image with a higher resolution.

According to an embodiment of the present invention, there is providedan image processor performing a super-resolution process of convertingan input image with a first resolution to an output image with a secondresolution higher than the first resolution with respect to a pluralityof input images which are consecutive in time, including: predictionmeans for predicting the output image with the second resolution of atime being processed using the input image of the time being processedand the output image obtained by performing the super-resolution processon an input image of a time earlier than the time being processed;generation means for generating a reduced image with the firstresolution composed of pixels at different phases of the predictionimage, the phases being different from time to time, using a predictionimage obtained by the prediction of the prediction means; differencecalculation means for calculating a difference between the input imageof the time being processed and the reduced image; and addition meansfor adding the difference which is up-sampled to the second resolutionto the prediction image, thus generating the output image with thesecond resolution of the time being processed.

The generation means may change a phase of each pixel of the predictionimage to be used for generating the reduced image for each time inaccordance with a predetermined pattern.

The generation means may include selection means for selecting a phaseof each pixel of the prediction image, and sampling means for generatingthe reduced image by sampling pixels at the phase selected by theselection means from the prediction image.

The generation means may include selection means for selecting a phaseof each pixel of the prediction image, and filtering means forgenerating pixels at the selected phase by a filtering process usingseveral pixels around a pixel of the prediction image which ispositioned at the phase selected by the selection means, thus generatingthe reduced image.

The generation means may include phase control means for generating thereduced image composed of pixels which are positioned at a phaseseparated by a predetermined distance in a predetermined direction froma predetermined reference phase of the prediction image, and shifting aphase of the difference up-sampled to the second resolution by thepredetermined distance in the predetermined direction.

The input image may be an image of an interlaced format. The generationmeans may include: switching means for changing an output destination ofthe prediction image depending on whether the input image of the timebeing processed is a top-field image or a bottom-field image; firstselection means for selecting a phase of each pixel of the predictionimage obtained from the input image of a top field which is output fromthe switching means; first sampling means for generating the reducedimage by sampling a pixel at the phase selected by the first selectionmeans from the prediction image; second selection means for selecting aphase of each pixel of the prediction image obtained from the inputimage of a bottom field which is output from the switching means; andsecond sampling means for generating the reduced image by samplingpixels at the phase selected by the second selection means from theprediction image.

The first and second selection means may independently change the phaseof each pixel of the prediction image used for generating the reducedimage from field to field in accordance with a predetermined pattern.

The generation means may include phase control means for moving eachpixel of the prediction image by a predetermined distance in apredetermined direction to shift a phase of each pixel of the predictionimage; and reduced image generation means for generating the reducedimage composed of pixels at a predetermined phase of the predictionimage in which the phase is shifted by the phase control means. Thephase control means may change a direction of shifting the phase of eachpixel of the prediction image in each time in accordance with apredetermined pattern.

According to another embodiment of the present invention, there isprovided an image processing method or program for performing asuper-resolution process of converting an input image with a firstresolution to an output image with a second resolution higher than thefirst resolution with respect to a plurality of input images which areconsecutive in time, the method including the steps of: predicting theoutput image with the second resolution of a time being processed usingthe input image of the time being processed and the output imageobtained by performing the super-resolution process on an input image ofa time earlier than the time being processed; generating a reduced imagewith the first resolution composed of pixels at different phases of theprediction image, the phases being different from time to time, using aprediction image obtained by the prediction of the prediction unit;calculating a difference between the input image of the time beingprocessed and the reduced image; and adding the difference which isup-sampled to the second resolution to the prediction image, thusgenerating the output image with the second resolution of the time beingprocessed.

According to the embodiment of the present invention, when performing asuper-resolution process of converting an input image with a firstresolution to an output image with a second resolution higher than thefirst resolution with respect to a plurality of input images which areconsecutive in time, the output image with the second resolution of atime being processed is predicted using the input image of the timebeing processed and the output image obtained by performing thesuper-resolution process on an input image of a time earlier than thetime being processed. Subsequently, a reduced image with the firstresolution composed of pixels at different phases of the predictionimage, the phases being different from time to time, is generated usinga prediction image obtained by the prediction of the prediction unit.Subsequently, a difference between the input image of the time beingprocessed and the reduced image is calculated. Lastly, the differencewhich is up-sampled to the second resolution is added to the predictionimage, whereby the output image with the second resolution of the timebeing processed is generated.

According to the embodiment of the present invention, it is possible toimprove further the image quality of images when an input image isconverted to an image with a higher resolution.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of an image processoraccording to related art.

FIG. 2 is a diagram illustrating the generation of a differential imagein the image processor of the related art.

FIG. 3 is a diagram illustrating the generation of a reduced imageaccording to the related art.

FIG. 4 is a diagram illustrating an example where the waveform of animage is preserved at the time of generating the reduced image accordingto the related art.

FIG. 5 is a diagram illustrating an example where the waveform of animage is not preserved at the time of generating the reduced imageaccording to the related art.

FIG. 6 is a diagram illustrating an outline of the processing of animage processor according to an embodiment of the present invention.

FIG. 7 is a diagram illustrating an outline of the processing of theimage processor according to the embodiment of the present invention.

FIG. 8 is a diagram illustrating an exemplary configuration of the imageprocessor according to the embodiment of the present invention.

FIG. 9 is a diagram illustrating the phases of the pixels used forgeneration of a reduced image.

FIG. 10 is a diagram illustrating changes in the phases of the pixelsused for the generation of the reduced image.

FIG. 11 is a flowchart illustrating an image conversion process.

FIG. 12 is a diagram illustrating another exemplary configuration of animage processor according to an embodiment of the present invention.

FIG. 13 is a diagram illustrating a filtering process performed at thetime of generating a reduced image.

FIG. 14 is a diagram illustrating a filtering process performed at thetime of generating a reduced image.

FIG. 15 is a diagram illustrating a filtering process performed at thetime of generating a reduced image.

FIG. 16 is a flowchart illustrating an image conversion process.

FIG. 17 is a diagram illustrating new errors generated at the time ofgenerating an SR image.

FIG. 18 is a diagram illustrating the suppression of the generation ofnew errors at the time of generation of the SR image.

FIG. 19 is a diagram illustrating another exemplary configuration of animage processor of an embodiment of the present invention.

FIG. 20 is a flowchart illustrating an image conversion process.

FIG. 21 is a diagram illustrating another exemplary configuration of animage processor according to an embodiment of the present invention.

FIG. 22 is a diagram illustrating the phases of the pixels used forgeneration of a reduced image.

FIG. 23 is a diagram illustrating changes in the phases of the pixelsused for the generation of the reduced image.

FIG. 24 is a flowchart illustrating an image conversion process.

FIG. 25 is a diagram illustrating another exemplary configuration of animage processor of an embodiment of the present invention.

FIG. 26 is a diagram illustrating a 2-dimensional filtering process.

FIG. 27 is a flowchart illustrating an image conversion process.

FIG. 28 is a diagram illustrating a shift in the phases of each pixel ofa combined image.

FIG. 29 is a diagram illustrating another exemplary configuration of animage processor according to an embodiment of the present invention.

FIG. 30 is a flowchart illustrating an image conversion process.

FIG. 31 is a diagram illustrating an exemplary configuration of acomputer.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the drawings.

Outline of Processing of Image Processor

First, an outline of the processing performed by an image processoraccording to an embodiment of the present invention will be described.

In the image processor, a time-consecutive image (hereinafter referredto as an LR image), such as a moving image, is input. In the imageprocessor, the LR image is converted to an image (hereinafter referredto as an SR image) with a higher resolution than the LR image by asuper-resolution process and output. The SR image is an enlarged imageof the LR image. The LR image may not be a moving image and may be aplurality of images which are time-consecutive and contain the samesubject captured at different times.

Specifically, the image processor predicts an SR image of the currentframe using an LR image of the current frame (time) being processed andan SR image obtained by the LR image of a previous frame (time) earlierthan the current frame and outputs an image obtained by prediction as acombined image. Then, the image processor down-samples the combinedimage to obtain a reduced image, up-samples a differential imageobtained by calculating a difference between the reduced image and theLR image, and adds the up-sampled differential image to the combinedimage, thus obtaining an SR image of the current frame.

According to the image processor, as illustrated in FIG. 6, when thereduced image is generated, the pixels at different phases of thecombined image are selected for each frame to be used for the generationof the reduced image, thus suppressing accumulation of noise in the SRimage. In FIG. 6, the vertical direction represents the pixel values ofpixels of an image, and the horizontal direction represents apredetermined direction of the image. In addition, each circlerepresents each pixel on the image.

Referring to FIG. 6, a waveform indicated by an arrow A51 represents thewaveform of a combined image of the current frame. That is, a curve thatconnects adjacent pixels (i.e., the pixel values thereof) of thecombined image forms the waveform of the combined image.

In this figure, when a value between the maximum value and the minimumvalue of the pixel value of the combined image is referred to as anintermediate value, the combined image is composed of pixels which arerepeatedly arranged in the horizontal direction in the figure in theorder of a pixel having the intermediate value, a pixel having themaximum value, a pixel having the intermediate value, and a pixel havingthe minimum value. That is to say, in the example of FIG. 6, thewaveform of the combined image has a sinusoidal shape.

Now, the pixels of the combined image are sampled every other pixel inthe horizontal direction so as to generate a reduced image.

For example, in the current frame, it will be assumed that the pixels ofthe combined image indicated by the arrow A51 are sampled every otherpixel in the rightward direction by starting with the third pixel fromthe left in the figure, whereby the pixels of the combined imageindicated by an arrow A52 are sampled and a reduced image is generated.In this case, the waveform of the obtained reduced image has a flatshape without any change in the horizontal direction of the figure, andthe original waveform of the combined image is not properly preserved.

Moreover, when the waveform of the LR image of the current frame has aflat shape without any change in the horizontal direction of the figureas indicated by an arrow A53, a differential image having a waveformindicated by an arrow A54 is obtained by calculating a differencebetween the reduced image and the LR image. The waveform of thedifferential image has a shape such that it is flat in the horizontaldirection of the figure, and errors generated in the combined image arenot detected in the differential image.

Therefore, although the differential image is up-sampled and added tothe combined image as a correction value of the errors of the combinedimage relative to the LR image, the errors, i.e., noise components arenot removed from the SR image obtained by addition. In the followingdescription, the direction in each image corresponding to the rightwarddirection of the figure will be also referred to as an x-direction.

The SR image obtained thus is used for generation of a combined image ofa subsequent frame. For example, it will be assumed that a combinedimage of a subsequent frame is generated using the SR image obtainedfrom the differential image indicated by the arrow A54, whereby acombined image having a waveform indicated by an arrow A55 is obtained.The combined image indicated by the arrow A55 has the same waveform asthe waveform of the combined image indicated by the arrow A51.

In the image processor, using the combined image indicated by the arrowA55, a reduced image which is composed of the pixels at a predeterminedphase of the combined image as indicated by an arrow A56 is generated.The dotted circles of the waveform of the combined image indicated bythe arrow A56 represent the positions of the pixels which are sampled atthe time of generating the reduced image in the previous frame. Thedotted circles represent the pixels at the same phase as the pixels ofthe combined image indicated by the arrow A51, which are used forgenerating the reduced image.

In the example indicated by the arrow A56, the left adjacent pixels inthe figure of the pixels of the combined image used at the time ofgenerating the reduced image of the previous frame are used forgeneration of a reduced image of the current frame being processed. Inthe image processor, the reduced image composed of such pixels isgenerated, and the waveform of the reduced image has the same shape asthe waveform of the combined image indicated by the arrow A55, namely, asinusoidal shape. In other words, in the reduced image, the originalwaveform of the combined image is properly preserved.

When the waveform of an LR image of a new frame to be processed has aflat shape without any change in the x-direction as indicated by anarrow A57, a differential image having a waveform indicated by an arrowA58 is obtained by calculating a difference between the reduced imageand the LR image. The waveform of the differential image has a shapesuch that the waveform of the combined image indicated by the arrow A55is reversed in the vertical direction of the figure with respect to thecentral position in the pixel value direction (vertical direction) ofthe waveform.

Since the original waveform of the combined image is not properlypreserved in the reduced image of this frame, the waveform of thedifferential image indicated by an arrow A58 has a shape which isindicative of an error between the combined image and the LR image.Therefore, when the differential image is added to the combined imageindicated by the arrow A55, an SR image having higher image quality isobtained in which errors (noise components) are removed.

Furthermore, the SR image is used for generating a combined image of asubsequent frame. Since noise components have been removed from the SRimage, the combined image obtained from the SR image will be a combinedimage having a waveform similar to the original waveform of the SR imageas indicated by an arrow A59, for example.

The waveform of the combined image indicated by the arrow A59 has a flatshape without any change in the x-direction. In the image processor, asindicated by an arrow A60, the pixels at the same phase as the pixels ofthe combined image of the frame indicated by the arrow A52, which areused for generating the reduced image, are used and a reduced imagecomposed of the pixels is generated. As a result, the obtained reducedimage will be an image having a waveform in which the original waveformof the combined image is properly preserved.

When the waveform of an LR image of a new frame to be processed has aflat shape without any change in the x-direction as indicated by anarrow A61, a differential image having a waveform indicated by an arrowA62 is obtained by calculating a difference between the reduced imageand the LR image. The waveform of the differential image has a flatshape without any change in the x-direction.

Since the original waveform of the combined image is properly preservedin the reduced image of this frame, when the differential image havingthe waveform indicated by the arrow A62 is added to the combined imageindicated by the arrow A59, an SR image having higher image quality isobtained in which noise components are not contained.

Furthermore, it will be assumed that from the SR image obtained thus, acombined image having a waveform having a shape such that it is flat inthe x-direction is obtained in a subsequent frame as indicated by anarrow A63. Then, as indicated by an arrow A64, a reduced image isgenerated which is composed of the pixels at a predetermined phase ofthe combined image indicated by the arrow A63. The dotted circles of thewaveform of the combined image indicated by the arrow A64 represent thepositions of the pixels of the combined image of the frame indicated bythe arrow A52, which are sampled at the time of generating the reducedimage.

In the combined image of the frame indicated by the arrow A64, the rightadjacent pixels in the figure of the pixels of the combined image usedfor generating the reduced image from the combined image of the frameindicated by the arrow A52 are used for generation of a reduced image.In the image processor, the reduced image composed of such pixels isgenerated, and the waveform of the reduced image has the same shape asthe waveform of the combined image indicated by the arrow A63.

When the waveform of an LR image of a new frame to be processed has aflat shape without any change in the x-direction as indicated by anarrow A65, a differential image having a waveform indicated by an arrowA66 is obtained by calculating a difference between the reduced imageand the LR image. The waveform of the differential image has a flatshape without any change in the x-direction.

Since the original waveform of the combined image is properly preservedin the reduced image of this frame, when the differential image havingthe waveform indicated by the arrow A66 is added to the combined imageindicated by the arrow A63, an SR image having higher image quality isobtained in which noise components are not contained.

In this way, by changing the phases of the pixels of the combined imageto be used for generating the reduced image from frame to frame, it ispossible to prevent accumulation of errors (noise components) generatedat a certain phase, whereby the image quality of a final SR image can beimproved.

When the phases of the pixels used for generation of the reduced imageare changed for each frame (time), there may occur a case, for example,where the noise in the combined image is temporarily amplified asillustrated in FIG. 7. In FIG. 7, the vertical direction represents thepixel values of pixels of an image, and the rightward directionrepresents the x-direction. In addition, each circle represents eachpixel on the image.

Referring to FIG. 7, a waveform indicated by an arrow A81 represents thewaveform of a combined image of the current frame. The waveform of thecombined image indicated by the arrow A81 has the same shape as thewaveform of the combined image indicated by the arrow A51 of FIG. 6.

Now, it will be assumed that the pixels of the combined image indicatedby the arrow A81 are sampled every other pixel by starting with thethird pixel from the left in the figure so as to generate a reducedimage in the x-direction indicated by an arrow A82. In this case, thewaveform of the obtained reduced image has a flat shape without anychange in the x-direction, and the original waveform of the combinedimage is not properly preserved.

Moreover, when the waveform of the LR image of the current frame has aflat shape without any change in the x-direction of the figure asindicated by an arrow A83, a differential image having a waveformindicated by an arrow A84 is obtained by calculating a differencebetween the reduced image and the LR image. The waveform of thedifferential image has a shape such that it is flat in the x-direction,and errors generated in the combined image are not detected in thedifferential image. Therefore, the errors, i.e., noise components arenot removed from the SR image obtained by addition of the differentialimage and the combined image.

The SR image obtained thus is used for generation of a combined image ofa subsequent frame. For example, it will be assumed that a combinedimage of a subsequent frame is generated using the SR image obtainedfrom the differential image indicated by the arrow A84, whereby acombined image having a waveform indicated by an arrow A85 is obtained.The combined image indicated by the arrow A85 has the same waveform asthe waveform of the combined image indicated by the arrow A81.

In the image processor, using the combined image indicated by the arrowA85, a reduced image which is composed of the pixels at a predeterminedphase of the combined image as indicated by an arrow A86 is generated.The dotted circles of the waveform of the combined image indicated bythe arrow A86 represent the positions of the pixels which are sampled atthe time of generating the reduced image in the previous frame. Thedotted circles represent the pixels at the same phase as the pixels ofthe combined image indicated by the arrow A81, which are used forgenerating the reduced image.

In the example indicated by the arrow A86, the left adjacent pixels inthe figure of the pixels of the combined image used at the time ofgenerating the reduced image of the previous frame are used forgeneration of a reduced image of a current frame being processed. In theimage processor, the reduced image composed of such pixels is generated,and the waveform of the reduced image has substantially the same shapeas the waveform of the combined image indicated by the arrow A85,namely, a sinusoidal shape. However, in this reduced image, the subjecton the combined image is shifted leftward in the figure by one pixelfrom the center of the reduced image.

When a difference between the reduced image obtained thus and an LRimage having a waveform as indicated by an arrow A87 having a shape suchthat it is flat in the x-direction is calculated, a differential imagehaving a waveform indicated by an arrow A88 is obtained. The waveform ofthe differential image has the same shape as the waveform of thecombined image indicated by the arrow A85.

Therefore, when the differential image is added to the combined imageindicated by the arrow A85, an SR image is obtained in which errors(noise components) are amplified. When a combined image of a subsequentframe is generated using such an SR image, a combined image having awaveform as indicated by an arrow A89 is obtained in which the noisecomponents of the combined image indicated by the arrow A85 areamplified.

Furthermore, in the image processor, as indicated by the arrow A90, areduced image is generated which is composed of the pixels at the samephase as the pixels of the combined image, which are used for generationof the reduced image in the frame indicated by the arrow A82. As aresult, the obtained reduced image will be an image having a waveform inwhich the original waveform of the combined image is not properlypreserved.

When a difference between the obtained reduced image and an LR imagehaving a waveform as indicated by an arrow A91 having a shape such thatit is flat in the x-direction is calculated, a differential image havinga waveform indicated by an arrow A92 is obtained. The waveform of thedifferential image has a shape such that it is flat in the x-direction.The differential image is added to a combined image indicated by anarrow A89 to obtain an SR image; however, the SR image will be an imagein which the amplified noise components are accumulated as they are.

Furthermore, it will be assumed that from this SR image, a combinedimage having a waveform having a sinusoidal shape as indicated by anarrow A93 is obtained in a subsequent frame. Then, as indicated by anarrow A94, a reduced image is generated which is composed of the pixelsat a predetermined phase of the combined image indicated by the arrowA93. The dotted circles of the waveform of the combined image indicatedby the arrow A94 represent the positions of the pixels of the combinedimage of the frame indicated by the arrow A82, which are sampled at thetime of generating the reduced image.

In the example indicated by the arrow A94, the left adjacent pixels inthe figure of the pixels of the combined image used for generating thereduced image from the combined image of the frame indicated by thearrow A82 are used for generation of a reduced image in the currentframe being processed. In the image processor, the reduced imagecomposed of such pixels is generated, and the waveform of the reducedimage has the same shape as the waveform of the combined image indicatedby the arrow A93.

When the waveform of an LR image of a new frame to be processed has aflat shape without any change in the x-direction as indicated by anarrow A95, a differential image having a waveform indicated by an arrowA96 is obtained by calculating a difference between the reduced imageand the LR image. The waveform of the differential image has a shapesuch that the waveform of the combined image indicated by the arrow A93is reversed in the vertical direction of the figure with respect to thecentral position in the pixel value direction of the waveform.

Since the original waveform of the combined image is properly preservedin the reduced image of this frame, the waveform of the differentialimage indicated by an arrow A96 has a shape which is indicative of anerror between the combined image and the LR image. Therefore, when thisdifferential image is added to the combined image indicated by the arrowA93, an SR image having higher image quality is obtained in which errors(noise components) are removed as indicated by an arrow A97. As seenfrom the waveform of the SR image indicated by the arrow A97, thewaveform has the same shape as the waveform of the LR image indicated bythe arrow A95, and the noise components which have been accumulatedwithout being detected are removed.

In this way, by changing the phases of the pixels of the combined imageto be used for generating the reduced image from frame to frame, thenoise components are removed by a repeated super-resolution process evenwhen errors are temporarily amplified and accumulated, whereby a finalSR image having a high quality can be obtained.

First Embodiment Configuration of Image Processor

Next, an embodiment of the image processor performing theabove-described super-resolution process will be described. FIG. 8 is adiagram illustrating an exemplary configuration of an image processoraccording to an embodiment of the present invention.

The image processor 61 includes an up-sampler 71, a motion vectordetector 72, a motion compensator 73, a mask generator 74, a mixer 75, aphase controller 76, down-samplers 77-1 to 77-3, a phase controller 78,down-samplers 79-1 to 79-3, a subtractor 80, an up-sampler 81, an adder82, and a buffer 83.

The image processor 61 receives an LR image of one frame to beprocessed, and the input LR image is supplied to the up-sampler 71 andthe subtractor 80.

The up-sampler 71 up-samples the supplied LR image to an image with thesame resolution as an SR image, which will be generated from now on, andsupplies the up-sampled image to the motion vector detector 72, the maskgenerator 74, and the mixer 75.

The motion vector detector 72 calculates a motion vector of the whole SRimage from the LR image supplied from the up-sampler 71 and an SR imageof a previous frame earlier than the current frame being processed,supplied from the buffer 83, and supplies the motion vector to themotion compensator 73.

The motion compensator 73 performs motion compensation using the motionvector supplied from the motion vector detector 72 and an SR imagesupplied from the buffer 83 to generate a prediction image.Specifically, the motion compensator 73 moves the whole SR image by adistance the same as the magnitude of the motion vector in a directionindicated by the motion vector, thus generating an image obtained thusas the prediction image. The prediction image is an image obtained bypredicting an SR image of the current frame by motion compensation usingthe SR image of the previous frame to the current frame. The predictionimage is supplied from the motion compensator 73 to the mask generator74 and the mixer 75.

The mask generator 74 generates a motion mask using the LR imagesupplied from the up-sampler 71 and the prediction image supplied fromthe motion compensator 73 and supplies the motion mask to the mixer 75.The motion mask is information used for specifying a region of the LRimage in which a moving subject is displayed, and which is generated bycalculating a difference between the LR image and the prediction image.

The mixer 75 performs weighted addition of the prediction image suppliedfrom the motion compensator 73 and the LR image supplied from theup-sampler 71 using the motion mask supplied from the mask generator 74and supplies a combined image obtained thus to the phase controller 76and the adder 82.

The phase controller 76 selects a phase of the combined image to be usedfor generating the reduced image and changes an output destination ofthe combined image supplied from the mixer 75 in accordance with theselected phase. That is to say, the phase controller 76 supplies thesupplied combined image to any one of the down-samplers 77-1 to 77-3.

Each of the down-samplers 77-1 to 77-3 samples the pixels at apredetermined phase of the combined image supplied from the phasecontroller 76 and generates an image composed of such pixels, thusdown-sampling the combined image in the x-direction. Each of thedown-samplers 77-1 to 77-3 supplies the generated image to the phasecontroller 78. In the following description, when it is not necessary todistinguish particularly between the down-samplers 77-1 to 77-3, theywill be simply referred to as a down-sampler 77.

The phase controller 78 selects a phase of the combined image to be usedfor generating the reduced image and changes an output destination ofthe image supplied from the down-sampler 77 in accordance with theselected phase. That is to say, the phase controller 78 supplies thesupplied image to any one of the down-samplers 79-1 to 79-3.

Each of the down-samplers 79-1 to 79-3 samples the pixels at apredetermined phase of the image supplied from the phase controller 78and generates a reduced image composed of such pixels. In this way, thecombined image is down-sampled in a direction (hereinafter referred toas a y-direction) perpendicular to the x-direction, and thus, a reducedimage with the same resolution as the LR image is obtained.

Each of the down-samplers 79-1 to 79-3 supplies the generated reducedimage to the subtractor 80. In the following description, when it is notnecessary to distinguish particularly between the down-samplers 79-1 to79-3, they will be simply referred to as a down-sampler 79.

The subtractor 80 subtracts the reduced image supplied from thedown-sampler 79 from the LR image supplied thereto to calculate adifference between the LR image and the reduced image and supplies adifferential image obtained thus to the up-sampler 81. The up-sampler 81up-samples the differential image supplied from the subtractor 80 to animage with the same resolution as the SR image and supplies an enlargedimage obtained thus to the adder 82.

The adder 82 adds the combined image supplied from the mixer 75 and theenlarged image supplied from the up-sampler 81 to generate an SR imageand supplies the SR image to a subsequent stage while supplying the SRimage to the buffer 83 to be stored therein. The buffer 83 stores the SRimage supplied from the adder 82 only for one frame period and suppliesthe SR image stored therein to the motion vector detector 72 and themotion compensator 73 when a subsequent frame is processed.

The down-sampler 77 and the down-sampler 79 are configured todown-sample the combined image by generating an image which is composedof the pixels at a predetermined phase (position) of the combined image.

For example, as illustrated in FIG. 9, the down-samplers 77 and 79sample pixels at different phases based on a predetermined referencephase. In FIG. 9, the rightward direction represents the x-direction,and the downward direction represents the y-direction. In addition, inthe figure, each circle represents each pixel on the combined image.

The down-sampler 77-1 samples a left adjacent pixel G12 of a referencepixel G11 which serves as a reference pixel on the combined image, asdepicted on the top left of the figure. Here, the reference pixel G11will be assumed as being a plurality of pixels which are arranged atpredetermined intervals on the combined image. For example, when thecombined image is down-sampled in the x-direction to half its pixelcount, a predetermined number of pixels on the combined image, which arearranged every other pixel in the x-direction, will be assumed as beingthe reference pixel G11.

The down-sampler 77-2 samples the reference pixel G11 on the combinedimage, as depicted on the top center of the figure. The down-sampler77-3 samples a right adjacent pixel G13 of the reference pixel G11 onthe combined image, as depicted on the top right of the figure.

The down-sampler 79-1 samples an upper adjacent pixel G22 of a referencepixel G21 which serves as a reference pixel on the combined image whichis down-sampled in the x-direction, as depicted on the bottom left ofthe figure. Here, the reference pixel G21 will be assumed as being aplurality of pixels which are arranged at predetermined intervals on thecombined image which is down-sampled in the x-direction. For example,when the combined image is down-sampled in the y-direction to half itspixel count, a predetermined number of pixels on the combined image,which are arranged every other pixel in the y-direction, will be assumedas being the reference pixel G21.

The down-sampler 79-2 samples the reference pixel G21 on the combinedimage, as depicted on the bottom center of the figure. The down-sampler79-3 samples a lower adjacent pixel G23 of the reference pixel G21 onthe combined image, as depicted on the bottom right of the figure.

Each of the phase controllers 76 and 78 selects for each frame as towhich one of the down-samplers 77 and 79 will be used for down-samplingthe combined image.

For example, as illustrated in FIG. 10, each of the phase controllers 76and 78 selects the output destination of the combined image so that thephases of the pixels to be sampled are changed in a predeterminedpattern.

In FIG. 10, the rightward direction represents time, and the numberabove each phase controller 76 represents a number that identifies theframe of the combined image to be processed. Specifically, the framesare processed in the order from the 0th frame indicated by “0” to the7th frame indicated by “7”. In addition, the bold line extending fromthe phase controller 76 to the down-sampler 79 represents a path alongwhich the combined image is supplied.

In the 0th frame which is first processed, the combined image issupplied from the phase controller 76 to the down-sampler 77-2, and thecombined image supplied from the down-sampler 77-2 to the phasecontroller 78 is then supplied to the down-sampler 79-2.

In the 1st frame subsequent to the 0th frame, the combined image issupplied from the phase controller 76 to the down-sampler 77-1, and thecombined image supplied from the down-sampler 77-1 to the phasecontroller 78 is then supplied to the down-sampler 79-2.

In this way, in the 0th frame and the 1st frame which are consecutive intime, a reduced image is generated which is composed of the pixels atdifferent phases (positions) of the combined image. Thus, accumulationof noise components (errors) is prevented.

Similarly, in the 2nd frame subsequent to the 1st frame, the combinedimage is supplied to the down-sampler 77-2 and the down-sampler 79-2.That is to say, the reference pixel G11 of the combined image is sampledby the down-sampler 77-2, and the reference pixel G21 of the combinedimage is sampled by the down-sampler 79-2, whereby a reduced image isgenerated.

In the 3rd frame, the combined image is supplied to the down-sampler77-2 and the down-sampler 79-1. In the 4th frame, the combined image issupplied to the down-sampler 77-2 and the down-sampler 79-2.

In the 5th frame, the combined image is supplied to the down-sampler77-3 and the down-sampler 79-2. In the 6th frame, the combined image issupplied to the down-sampler 77-2 and the down-sampler 79-2. In the 7thframe, the combined image is supplied to the down-sampler 77-2 and thedown-sampler 79-3.

In the 8th and later frames, the same pattern as the 0th to 7th framesis repeated, whereby the combined image is supplied to each down-sampler77 and each down-sampler 79. Therefore, in the (1+8i)th frame (where iis a natural number), the combined image is supplied to the down-sampler77-1 and the down-sampler 79-2.

When the amount of shifting in the phase of the pixels of the combinedimage used for generation of the reduced image is increased, the amountof a positional shift between the position of the subject in the reducedimage and the position of the subject in the combined image willincrease, whereby new errors (noise components) different from theerrors generated at the time of generating the combined image will begenerated in the SR image. Therefore, it is preferable that the amountof shifting in the phase of the pixels of the combined image is set assmall as possible.

The phases of the pixels of the combined image to be used for generatingthe reduced image may be changed in a predetermined pattern and may berandomly selected for each frame.

Operation of Image Processor

Next, the operation of the image processor 61 described above will bedescribed. When an LR image is supplied to the image processor 61, theimage processor 61 starts an image conversion process which is a processof converting the LR image of each frame to an SR image by asuper-resolution process. Hereinafter, by referring to the flowchart ofFIG. 11, the image conversion process by the image processor 61 will bedescribed.

At step S11, the up-sampler 71 up-samples an LR image supplied theretoto an image with the same resolution as an SR image, which will begenerated from now on, and supplies the up-sampled image to the motionvector detector 72, the mask generator 74, and the mixer 75.

At step S12, the motion compensator 73 performs motion compensation togenerate a prediction image.

Specifically, the motion vector detector 72 calculates one motion vectorof the whole SR image from the LR image supplied from the up-sampler 71and an SR image supplied from the buffer 83 and supplies the motionvector to the motion compensator 73. Moreover, the motion compensator 73moves the whole SR image by a distance the same as the magnitude of themotion vector in a direction indicated by the motion vector suppliedfrom the motion vector detector 72 to generate a prediction image andsupplies the generated prediction image to the mask generator 74 and themixer 75.

At step S13, the mask generator 74 generates a motion mask using the LRimage supplied from the up-sampler 71 and the prediction image suppliedfrom the motion compensator 73 and supplies the motion mask to the mixer75.

For example, the mask generator 74 sequentially uses each pixel of amotion mask, which will be obtained from now on, as a target pixel, andcalculates a difference between the pixel value of a pixel of the LRimage and the pixel value of a pixel of the prediction image, the pixelsbeing located at the same position as the target pixel. The maskgenerator 74 uses the pixel value of the target pixel to a value whichis determined with respect to the absolute value of the calculateddifference. For example, the pixel value of the target pixel has alarger value as the absolute value of the difference decreases.

Therefore, in a region of the motion mask where the pixel value of apixel is small, the difference between the LR image and the predictionimage becomes larger, and the errors in the prediction based on themotion compensation become larger. In other words, a region of themotion mask where the pixel value of a pixel is large can be said to bea region of the LR image where the subject shows a motion different fromthe motion of the whole LR image; that is, a region where the subject ismoving relative to the whole LR image.

At step S14, the mixer 75 performs weighted addition of the predictionimage supplied from the motion compensator 73 and the LR image suppliedfrom the up-sampler 71 using the motion mask supplied from the maskgenerator 74 to generate a combined image and supplies the combinedimage to the phase controller 76 and the adder 82.

For example, the mixer 75 sequentially uses each pixel of a combinedimage, which will be obtained from now on, as a target pixel. Then,based on the pixel value of the pixel of the motion mask at the sameposition as the target pixel, the mixer 75 calculates a weighting factorWi (0≦W≦1) of the pixel of the LR image at the same position as thetarget pixel and a weighting factor Wj (=1−Wi) of the pixel of theprediction image at the same position as the target pixel.

The mixer 75 uses, as the pixel value of the target pixel, a valueobtained by the addition of the pixel value of the pixel of the LR imageat the same position as the target pixel, multiplied by the weightingfactor Wi, and the pixel value of the pixel of the prediction image atthe same position as the target pixel, multiplied by the weightingfactor Wj. In this way, the mixer 75 uses each pixel of the combinedimage as the target pixel and calculates the pixel values of suchpixels, thus generating a combined image.

When performing weighted addition of the LR image and the predictionimage, the weighting factor Wi is set to be larger as the pixel value ofthe pixel of the motion mask decreases. This is to increase thecontribution ratio of the LR image to the generation of the combinedimage in the region where the pixel value of the pixel of the motionmask is small, namely a region where the accuracy of the predictionbased on the motion compensation is low, thus preventing image qualitydeterioration of the combined image.

At step S15, the phase controller 76 selects the x-directional phase ofthe combined image to be used for generating a reduced image. Forexample, when the x-directional phase is changed in the patterndescribed with reference to FIG. 10, the phase controller 76 selects thephase based on the frame number in FIG. 10 of the current frame beingprocessed.

The phase controller 76 supplies the combined image supplied from themixer 75 to the down-sampler 77 corresponding to the selected phase andcauses x-directional down-sampling to be performed. For example, whenthe current frame corresponds to the 5th frame in FIG. 10, the combinedimage is supplied to the down-sampler 77-3.

At step S16, the down-sampler 77 performs x-directional down-sampling onthe combined image supplied from the phase controller 76 and supplies animage obtained thus to the phase controller 78. Specifically, thedown-sampler 77 samples the pixels at a predetermined phase of thesupplied combined image to generate an image composed of the sampledpixels.

At step S17, the phase controller 78 selects the y-directional phase ofthe combined image to be used for generating a reduced image. Forexample, when the y-directional phase is changed in the patterndescribed with reference to FIG. 10, the phase controller 78 selects thephase based on the frame number in FIG. 10 of the current frame beingprocessed.

The phase controller 78 supplies the combined image supplied from thedown-sampler 77 to the down-sampler 79 corresponding to the selectedphase and causes y-directional down-sampling to be performed.

At step S18, the down-sampler 79 performs y-directional down-sampling onthe combined image supplied from the phase controller 78 and supplies animage obtained thus to the subtractor 80. Specifically, the down-sampler79 samples the pixels at a predetermined phase of the supplied combinedimage to generate a reduced image composed of the sampled pixels.

In this way, by down-sampling the combined image in the x andy-directions, a reduced image with the same resolution as the LR imageis generated, and the generated reduced image is supplied to thesubtractor 80.

At step S19, the subtractor 80 calculates a difference between the LRimage supplied thereto and the reduced image supplied from thedown-sampler 79 to generate a differential image. Specifically, thepixel values of the pixels of the differential image are used as thevalues of the difference between the pixel values of the pixels of theLR image and the pixel values of the pixels of the reduced image at thesame positions as the pixels.

The pixel values of the pixels of the differential image generated thusrepresent the difference between the reduced image obtained bypredicting the LR image (SR image) and the real LR image. Therefore,when the combined image is corrected by the amount of the difference, anSR image will be obtained in which the LR image is more faithfullyenlarged; that is to say, an SR image will be obtained in which thewaveform of the LR image is preserved as it is. The subtractor 80supplies the generated differential image to the up-sampler 81.

At step S20, the up-sampler 81 up-samples the differential imagesupplied from the subtractor 80 to an image with the same resolution asthe SR image and supplies an enlarged image obtained thus to the adder82.

At step S21, the adder 82 adds the combined image supplied from themixer 75 and the enlarged image supplied from the up-sampler 81 togenerate an SR image of the current frame and supplies the SR image to asubsequent stage while supplying the SR image to the buffer 83 to bestored therein.

At step S22, the image processor 61 determines whether or not theprocess should be terminated. For example, the process is determined tobe terminated when the supply of the LR image to the image processor 61stops and a process termination command is received.

When it is determined at step S22 that the process should not beterminated, the process returns to step S11 and the above-describedprocesses are repeated. That is to say, the LR image of a subsequentframe is used as the LR image of the current frame being processed andconverted to an SR image by a super-resolution process.

When it is determined at step S22 that the process should be terminated,each part of the image processor 61 terminates its pending processes,and the image conversion process ends.

In this way, the image processor 61 performs a super-resolution processon the input LR image and converts the LR image to the SR image. Inparticular, when generating the reduced image by down-sampling thecombined image obtained by prediction, the image processor 61 changesthe phases of the pixels to be used for generating the reduced imagefrom frame to frame.

As described above, since the phases of the pixels of the combined imageto be used for generating the reduced image are changed slightly fromframe to frame, it is possible to prevent accumulation of errors (noisecomponents) generated at a certain phase in the SR image, whereby theimage quality of the SR image can be improved.

Furthermore, since the phases of the pixels to be used for generatingthe reduced image are shifted slightly for each frame in accordance witha predetermined pattern, it is not necessary to specify the phase of thepixels of the combined image so that the original waveform of thecombined image is preserved in the reduced image. Therefore, it ispossible to obtain the SR image more quickly.

Second Embodiment Configuration of Image Processor

Although it has been described that the reduced image is generated bysampling the pixels of the combined image, the reduced image may begenerated by performing a filtering process on the combined image.

In such a case, the image processor may be configured as illustrated inFIG. 12, for example. In FIG. 12, the same or corresponding portions asthose in FIG. 8 will be denoted by the same reference numerals, anddescription thereof will be appropriately omitted.

The image processor 121 of FIG. 12 includes a phase controller 131,filtering processors 132-1 to 132-3, a phase controller 133, andfiltering processors 134-1 to 134-3, in lieu of the phase controller 76,the down-samplers 77-1 to 77-3, the phase controller 78, and thedown-samplers 79-1 to 79-3 of the image processor 61.

The phase controller 131 selects the phase of the combined image to beused for generating the reduced image and changes an output destinationof the combined image supplied from the mixer 75 in accordance with theselected phase. That is to say, the phase controller 131 supplies thesupplied combined image to any one of the filtering processors 132-1 to132-3.

Each of the filtering processors 132-1 to 132-3 calculates the pixels ata predetermined phase of the combined image supplied from the phasecontroller 131 by a filtering process and generates an image composed ofthe calculated pixels, thus down-sampling the combined image in thex-direction. Each of the filtering processors 132-1 to 132-3 suppliesthe generated image to the phase controller 133. In the followingdescription, when it is not necessary to distinguish particularlybetween the filtering processors 132-1 to 132-3, they will be simplyreferred to as a filtering processor 132.

The phase controller 133 selects a phase of the combined image to beused for generating the reduced image and changes an output destinationof the combined image supplied from the filtering processor 132 inaccordance with the selected phase. That is to say, the phase controller133 supplies the supplied combined image to any one of the filteringprocessors 134-1 to 134-3.

Each of the filtering processors 134-1 to 134-3 calculates the pixels ata predetermined phase of the combined image supplied from the phasecontroller 133 by a filtering process and generates a reduced imagecomposed of the calculated pixels, thus down-sampling the combined imagein the y-direction. Each of the filtering processors 134-1 to 134-3supplies the generated reduced image to the subtractor 80. In thefollowing description, when it is not necessary to distinguishparticularly between the filtering processors 134-1 to 134-3, they willbe simply referred to as a filtering processor 134.

As described above, in the filtering processor 132 or 134, the pixelvalues of the pixels are calculated by a filtering process. Therefore,the phase of the combined image selected by the phase controller 131 or133 is not necessarily the position of the pixel on the combined image,but the phase may be shifted with an accuracy of a pixel or lower.

For example, it will be assumed that the filtering processor 132-2calculates pixels at the phase of the reference pixel of the combinedimage, and the filtering processors 132-1 and 132-3 calculate pixels ata phase shifted by ½ pixel from the reference pixel of the combinedimage. More specifically, although no pixel exists at positions of thecombined image separated by a distance of a pixel of smaller from thereference pixel, it is assumed that pixels exist at such positions, andthe pixel values of the pixels are calculated.

In such a case, the filtering processor 132-2 calculates pixels at thephase of the reference pixel by a 3-tap filtering process as illustratedin FIG. 13. In FIG. 13, the rightward direction represents thex-direction.

The filtering processor 132-2 multiplies three coefficients W1 to W3 asindicated by an arrow Q11 with the pixel values of three pixels aroundthe reference pixel arranged in the x-direction on the combined image,respectively, thus calculating the sum of the pixel values. Among thecoefficients indicated by the arrow Q11, the coefficient W2 represents acoefficient multiplied with the reference pixel, and the coefficients W1and W3 represent coefficients multiplied with the adjacent pixels of thereference pixel, respectively.

In FIG. 13, the vertical direction represents the magnitude of eachcoefficient, and the coefficient has a larger value as it is located atthe higher position. Specifically, the coefficients W1 and W3 have thesame value, and the coefficients W1 and W3 have the smaller value thanthe coefficient W2.

The filtering processor 132-2 multiplies the coefficients W1 to W3 withthe pixel values of the three pixels arranged in the x-direction tocalculate a pixel value as indicated by an arrow Q12. In the exampleindicated by the arrow Q12, each circle represents each pixel. Inaddition, in the figure, the upper line of circles arranged in thehorizontal direction (x-direction) in the figure represents a combinedimage, and the lower line of circles arranged in the horizontaldirection in the figure represents an image obtained by performingx-directional down-sampling.

For example, when it is assumed that a pixel G42 positioned at thecenter of the pixels G41 to G43 on the combined image, which arearranged adjacent to each other in the x-direction, is a referencepixel, the pixel value of a pixel P11 is calculated using the pixels G41to G43. This pixel P11 is a pixel of the combined image which will becalculated from now on, the combined image being down-sampled in thex-direction, and the pixel P11 is positioned at the same phase as thepixel G42 which is the reference pixel.

The filtering processor 132-2 multiplies the coefficients W1 to W3,respectively, with the pixel values of the pixels G41 to G43, calculatesthe sum of the pixel values multiplied with the coefficients W1 to W3,and uses a value obtained thus as the pixel value of the pixel P11.

Each of the filtering processors 132-1 and 132-3 calculates the pixel ata phase shifted by ½ phase from the reference pixel by a 4-tap filteringprocess as illustrated in FIG. 14. In FIG. 14, the rightward directionrepresents the x-direction.

Each of the filtering processors 132-1 and 132-3 multiplies fourcoefficients W11 to W14 as indicated by an arrow Q21 with the pixelvalues of four pixels including the reference pixel arranged in thex-direction on the combined image, respectively, thus calculating thesum of the pixel values.

In FIG. 14, the vertical direction represents the magnitude of eachcoefficient, and the coefficient has a larger value as it is located atthe higher position.

The coefficients W12 and W13 which are multiplied with adjacent pixelshave the same value. The coefficients W11 and W14 which are multipliedwith pixels located so as to surround the pixels multiplied with thecoefficients W12 and W13 have the same value, and the coefficients W11and W14 have the smaller value than the coefficients W12 and W13.

The filtering processor 132-1 multiplies the coefficients W11 to W14with the pixel values of the four pixels arranged in the x-direction tocalculate a pixel value as indicated by an arrow Q22. In the exampleindicated by the arrow Q22 and an arrow Q23 to be described, each circlerepresents each pixel. In addition, the upper line of circles arrangedin the horizontal direction (x-direction) in the figure represents acombined image, and the lower line of circles arranged in the horizontaldirection in the figure represents an image obtained by performingx-directional down-sampling.

For example, in the example indicated by the arrow Q22, when it isassumed that a pixel G53 positioned approximately at the center of thepixels G51 to G54 on the combined image, which are arranged adjacent toeach other in the x-direction, is a reference pixel, the pixel value ofa pixel P21 is calculated using the pixels G51 to G54. This pixel P21 isa pixel of the combined image which will be calculated from now on, thecombined image being down-sampled in the x-direction, and the pixel P21is positioned at a position (phase) separated leftward (in the oppositedirection of the x-direction) in the figure by a distance of ½ pixelfrom the pixel G53 which is the reference pixel.

The filtering processor 132-1 multiplies the coefficients W11 to W14,respectively, with the pixel values of the pixels G51 to G54, calculatesthe sum of the pixel values multiplied with the coefficients W11 to W14,and uses a value obtained thus as the pixel value of the pixel P21.

The filtering processor 132-3 multiplies the coefficients W11 to W14with the pixel values of the four pixels arranged in the x-direction tocalculate a pixel value as indicated by an arrow Q23.

For example, when it is assumed that a pixel G53 positionedapproximately at the center of the pixels G52 to G55 on the combinedimage, which are arranged adjacent to each other in the x-direction, isa reference pixel, the pixel value of a pixel P22 is calculated usingthe pixels G52 to G55. This pixel P22 is a pixel of the combined imagewhich will be calculated from now on, the combined image beingdown-sampled in the x-direction, and the pixel P22 is positioned at aposition (phase) separated rightward (in the x-direction) in the figureby a distance of ½ pixel from the pixel G53 which is the referencepixel.

The filtering processor 132-3 multiplies the coefficients W11 to W14,respectively, with the pixel values of the pixels G52 to G55, calculatesthe sum of the pixel values multiplied with the coefficients W11 to W14,and uses a value obtained thus as the pixel value of the pixel P22.

As described above, the filtering processor 132 performs the filteringprocess by using different pixels depending on whether it calculates apixel at a phase shifted leftward or rightward from the reference pixel.

Specifically, when a pixel at a phase shifted leftward from thereference pixel is calculated, a pixel at a desired phase is calculatedusing the reference pixel, two left adjacent pixels of the referencepixel, and one right adjacent pixel of the reference pixel in thefigure. When a pixel at a phase shifted leftward from the referencepixel is calculated, a pixel at a desired phase is calculated using thereference pixel, one left adjacent pixel of the reference pixel, and tworight adjacent pixels of the reference pixel in the figure.

Similar to the case of the filtering processor 132 described withreference to FIGS. 13 and 14, the filtering processor 134 calculates apixel at a desired phase using pixels arranged consecutively in they-direction.

For example, the filtering processor 134-2 calculates the pixel value ofa pixel at the same phase as the reference pixel of the combined imageby a filtering process. The filtering processor 134-1 calculates thepixel value of a pixel at a phase (position) separated by apredetermined distance in the opposite direction of the y-direction fromthe reference pixel of the combined image by a filtering process. Thefiltering processor 134-3 calculates the pixel value of a pixel at aphase (position) separated by a predetermined distance in they-direction from the reference pixel of the combined image by afiltering process.

The phase of the pixel calculated by the filtering processor 132-1 or132-3 is not limited to a position shifted by ½ pixel from the referencepixel but may be any position. For example, as illustrated in FIG. 15, apixel at a position shifted by ¼ pixel from the reference pixel may begenerated. In FIG. 15, the rightward direction represents thex-direction.

In general, when a pixel at a phase shifted rightward in the figure by ¼phase from the reference pixel is calculated, each pixel of the combinedimage is first shifted rightward in the figure by ¼ pixel as indicatedby an arrow Q31, and then, a filtering process is performed. In theexample indicated by the arrow Q31, each circle in the figure representsone pixel. In addition, the upper line of circles represents a combinedimage, and the lower line of circles represents an image obtained byshifting the phase of each pixel of the combined image rightward in thefigure by ¼ pixel.

When the phase of each pixel of the combined image is shifted in thismanner, the pixel values of the three pixels of the phase-shiftedcombined image which are arranged in the x-direction are multiplied withcoefficients W21 to W23 indicated by an arrow Q32, and the sum of thepixel values multiplied with the coefficients is calculated. The valueof the calculated sum is used as the pixel value of the pixel at theposition shifted rightward in the figure by ¼ pixel from the referencepixel.

In the example indicated by the arrow Q32 and an arrow Q33 to bedescribed, the vertical direction in the figure represents the magnitudeof each coefficient, and the coefficient has a larger value as it islocated at the higher position.

In the example indicated by the arrow Q32, the coefficient W22represents a coefficient multiplied with the phase-shifted referencepixel, and the coefficients W21 and W23 represent coefficientsmultiplied with the horizontally adjacent pixels of the phase-shiftedreference pixel.

As described above, it is generally necessary to perform the filteringprocess after shifting the phase of the combined image. However, byusing a 4-tap filtering process as indicated by an arrow Q33 withincreased number of taps used in the filtering process, it is possibleto calculate a pixel at a desired phase with simpler processing. That isto say, by just adjusting the coefficient of each tap appropriately, itis not necessary to generate an image obtained by shifting the phases ofthe combined image before the filtering process.

In the example indicated by the arrow Q33, the pixel values of fourpixels, which include the reference pixel of the combined image and arearranged in the x-direction, are multiplied with coefficients W31 toW34, and the sum of the pixel values multiplied with the coefficients isused as the pixel value of a pixel which is to be calculated. That is tosay, the sum is used as the pixel value of the pixel at a positionshifted by ¼ pixel in the x-direction from the reference pixel.

The coefficients have different values which can be arranged in thedescending order of W32, W33, W31, and W34. When the distance or thedirection of the pixel to be calculated, from the reference pixel ischanged, the values of the coefficients W31 to W34 may be changed.

As described above, according to the filtering processors 132 and 134,it is possible to calculate the pixel value of a pixel at a phaseseparated by a certain distance in a certain direction from thereference pixel of the combined image by a filtering process withoutshifting the phase of each pixel of the combined image.

Operation of Image Processor

Next, with reference to the flowchart of FIG. 16, the image conversionprocess by the image processor 121 will be described. In FIG. 16, theprocesses of steps S51 to S54 are the same as the processes of steps S11to S14 in FIG. 11, and description thereof will be omitted.

At step S55, the phase controller 131 selects the x-directional phase ofthe combined image to be used for generating a reduced image. Forexample, when the x-directional phase is changed in a predeterminedpattern for each frame, the phase controller 131 selects a phasespecified by the pattern and the current frame being processed.

The phase controller 131 supplies the combined image supplied from themixer 75 to the filtering processor 132 corresponding to the selectedphase and causes x-directional down-sampling to be performed.

At step S56, the filtering processor 132 performs x-directionaldown-sampling on the combined image supplied from the phase controller131 by a filtering process and supplies an image obtained thus to thephase controller 133.

For example, the filtering processor 132 multiplies the pixel value ofthree or four pixels, which include the reference pixel on the combinedimage and are arranged consecutively in the x-direction, withpredetermined coefficients and calculates the sum of the pixel valuesmultiplied with the coefficients. Then, the filtering processor 132 usesthe value of the calculated sum as the value of the pixel value of thepixel at the selected phase of the combined image and generates an imagecomposed of the pixels at the selected phase of the combined image, thusdown-sampling the combined image.

At step S57, the phase controller 133 selects the y-directional phase ofthe combined image to be used for generating a reduced image. Forexample, when the y-directional phase is changed in a predeterminedpattern for each frame, the phase controller 133 selects a phasespecified by the pattern and the current frame being processed.

The phase controller 133 supplies the combined image supplied from thefiltering processor 132 to the filtering processor 134 corresponding tothe selected phase and causes y-directional down-sampling to beperformed.

At step S58, the filtering processor 134 performs y-directionaldown-sampling on the combined image supplied from the phase controller133 by a filtering process and supplies an image obtained thus to thesubtractor 80.

For example, the filtering processor 134 multiplies the pixel value ofthree or four pixels, which include the reference pixel on the combinedimage and are arranged consecutively in the y-direction, withpredetermined coefficients and calculates the sum of the pixel valuesmultiplied with the coefficients. Then, the filtering processor 134 usesthe value of the calculated sum as the value of the pixel value of thepixel at the selected phase of the combined image and generates areduced image composed of the pixels at the selected phase of thecombined image, thus down-sampling the combined image.

In this way, by down-sampling the combined image in the x andy-directions, a reduced image with the same resolution as the LR imageis generated, and the generated reduced image is supplied to thesubtractor 80. Subsequently, the processes of steps S59 to S62 areperformed, and the image conversion process ends. Since such processesare the same as the processes of steps S19 to S22 of FIG. 11,description thereof will be omitted.

In this way, the image processor 121 performs a super-resolution processon the input LR image and converts the LR image to the SR image. Inparticular, when generating the reduced image from the combined imageobtained by prediction by the filtering process, the image processor 121generates the reduced image so that the reduced image is composed ofpixels at different phases of the combined image for each frame.

As described above, since the reduced image is generated by thefiltering process, it is not only possible to shift the phases of thepixels of the combined image used for generating the reduced image withan accuracy of a pixel or lower, but also to decrease further the amountof a phase shift in the subject between the reduced image and thecombined image. In this way, it is possible to decrease further theerrors caused by the phase shift in the subject generated at the time ofgenerating the SR image. That is to say, it is possible to detect moreaccurately the errors between the combined image and the LR image andimprove the image quality of the SR image. Moreover, it is possible toobtain the reduced image more simply and quickly by the filteringprocess.

Furthermore, since the phases of the pixels to be used for generatingthe reduced image are changed from frame to frame, it is possible toprevent accumulation of errors (noise components) generated at the timeof generating the combined image, whereby the image quality of the SRimage can be improved.

Third Embodiment Configuration of Image Processor

When the amount of shifting in the phase of the pixels of the combinedimage used for generation of the reduced image is increased, the amountof a positional shift between the position of the subject in the reducedimage and the position of the subject in the combined image willincrease, whereby new errors (noise components) will be generated in thedifferential image as illustrated in FIG. 17. In FIG. 17, the verticaldirection represents the pixel values of pixels of an image, and therightward direction represents the x-direction in the image. Inaddition, each circle in the figure represents one pixel on the image.

Referring to FIG. 17, a waveform indicated by an arrow A121 representsthe waveform of a combined image of the current frame. That is, a curvethat connects adjacent pixels of the combined image indicated by thearrow A121 forms the waveform of the combined image. In the combinedimage, the pixel values of the pixels located slightly close to theright side from the center of the combined image are larger than thepixel values of other pixels and are protruding upward in the figure.

Now, the pixels of the combined image are sampled every other pixel inthe x-direction in the figure so as to generate a reduced image. Forexample, when the sampling positions of the pixels used for generatingthe reduced image are shifted by 2 pixels rightward in the figure from areference pixel, the position of a subject in the reduced image will beshifted by 2 pixels leftward in the figure from the center of the imageas indicated by an arrow A122, when compared with the position of thesubject on the combined image.

Moreover, when the waveform of the LR image of the current frame has aflat shape without any change in the x-direction as indicated by anarrow A123, a differential image having a waveform indicated by an arrowA124 is obtained by calculating a difference between the reduced imageand the LR image. When the differential image is up-sampled to an imagewith the same resolution as the SR image, an enlarged image having awaveform indicated by an arrow A125 is obtained. The waveform of theenlarged image has a shape such that the pixel values of the pixelslocated slightly close to the left side from the center of the enlargedimage are smaller than the pixel values of other pixels and areprotruding downward in the figure.

When the enlarged image is added to the combined image indicated by thearrow A121, a SR image having a waveform indicated by an arrow A126 isobtained. In the SR image, some errors remain unremoved from thecombined image and new errors generated by the addition of the enlargedimage are included.

That is to say, although errors are corrected at the center of the SRimage in the figure, the errors (noise components) included in thecombined image are not removed in a portion of the SR image which islocated slightly close to the right side from the center of the SRimage, but remain as they are. That is to say, in an ideal case, thewaveform of the SR image should have a shape such that it is flat in thex-direction like the LR image indicated by the arrow A123. However, theportion of the SR image which in the figure is located slightly close tothe right side from the center protrude upward in the figure like thecombined image indicated by the arrow A121.

Furthermore, in a portion of the SR image which in the figure is locatedslightly close to the left side from the center, new errors (noisecomponents) generated by the addition of the enlarged image to thecombined image are generated. That is to say, the portion of the SRimage, which in the figure is located slightly close to the left sidefrom the center, protrudes downward in the figure.

As described above, when the phases of the pixels of the combined imageused for generating the reduced image are shifted too much, new noisesare generated in the SR image.

Therefore, as illustrated in FIG. 18, by shifting each pixel of theenlarged image in the opposite direction of the direction of shiftingthe subject in the reduced image by the same shift amount, it ispossible to remove more certainly the errors included in the combinedimage and suppress generation of new errors.

In FIG. 18, the vertical direction represents the pixel values of pixelsof an image, and the rightward direction represents the x-direction inthe image. Moreover, in the figure, each circle represents each pixel onthe image. Furthermore, arrows A131 to A135 represent the waveforms of acombined image, a reduced image, an LR image, a differential image, andan enlarged image, respectively, and these waveforms are the same as thecase indicated by the arrows A121 to A125 in FIG. 17, and descriptionthereof will be omitted.

In the example of FIG. 18, a reduced image having a waveform indicatedby the arrow A132 is obtained from a combined image having a waveformindicated by the arrow A131. A difference between the reduced image andan LR image having a waveform indicated by the arrow A133 is calculated,whereby a differential image having a waveform indicated by the arrowA134 is obtained. When the differential image is up-sampled, an enlargedimage having a waveform indicated by the arrow A135 is obtained.

As indicated by the arrow A136, the pixels of the enlarged image areshifted rightward in the figure, whereby a final enlarged image isobtained.

That is to say, in the example of FIG. 18, as described above withreference to FIG. 17, when the reduced image is generated, since thepositions of the pixels sampled from the combined image are shiftedrightward in the figure by 2 pixels from the reference pixel, a phaseshift of 2 pixels is generated leftward in the enlarged image.Therefore, when the phase of each pixel of the enlarged image is shiftedby 2 pixels rightward in the figure, it is possible to correct a phaseshift in each pixel and cancel the generated phase shift.

In this way, when the phases of the pixels of the enlarged image areshifted in the direction of shifting the phases of the pixels of thecombined image used for generating the reduced image by the same shiftdistance and the phase-adjusted enlarged image is added to the combinedimage, it is possible to obtain a SR image with fewer errors asindicated by the arrow A137.

That is to say, by adding the enlarged image indicated by the arrow A136to the combined image indicated by the arrow A131, it is possible toobtain a SR image in which the noises included in the combined image areremoved and new noises are not included as indicated by the arrow A137.The waveform of the SR image indicated by the arrow A137 has a shapesuch that it is the same shape as the waveform of the LR image indicatedby the arrow A133 and it is flat in the x-direction.

As described above, the image processor adjusting the phases of thepixels of the enlarged image is configured as illustrated in FIG. 19,for example. In FIG. 19, the same or corresponding portions as those inFIG. 8 will be denoted by the same reference numerals, and descriptionthereof will be appropriately omitted.

The image processor 161 of FIG. 19 further includes a phase controller171 in the image processor 61 of FIG. 8. The phase controller 171 shiftsthe phase of each pixel of the enlarged image supplied from theup-sampler 81 in the direction of shifting the phase by the down-sampler77 or 79 by the same shift distance and supplies an image obtained thusto the adder 82.

Operation of Image Processor

Next, with reference to the flowchart of FIG. 20, the image conversionprocess by the image processor 161 will be described. The processes ofsteps S91 to S100 are the same as the processes of steps S11 to S20 ofFIG. 11, and description thereof will be omitted.

At step S101, the phase controller 171 shifts the phase of each pixel ofthe enlarged image supplied from the up-sampler 81 and supplies an imageobtained thus to the adder 82.

For example, it will be assumed that when the reduced image isgenerated, the down-sampler 77-3 generates an image composed of pixelson the combined image which are located at positions separated by 2pixels in the x-direction from the reference pixel. Moreover, it will beassumed that the down-sampler 79-3 generates a reduced image composed ofpixels on the image generated by the down-sampler 77-3, which arelocated at positions separated by 2 pixels in the y-direction from thereference pixel.

In such a case, the phase controller 171 moves each pixel of theenlarged image by a distance of 2 pixels in the x and y-directions anduses an image obtained thus as a final enlarged image. That is to say,the pixel values of the pixels of the final enlarged image obtainedafter the phase adjustment will have the pixel values of pixels whichare located at positions separated by 2 pixels in the opposite directionof the x-direction and by 2 pixels in the opposite direction of they-direction from the pixels of the enlarged image before the phaseadjustment, which are located at the same position.

At step S102, the adder 82 adds the enlarged image supplied from thephase controller 171 to the combined image supplied from the mixer 75 togenerate a SR image. Subsequently, the process of step S103 isperformed, and the image conversion process ends. Since the process ofstep S103 is the same as the process of step S22 of FIG. 11, descriptionthereof will be omitted.

In this way, the image processor 161 shifts the phase of the enlargedimage obtained by up-sampling the differential image and then adds thephase-shifted enlarged image to the combined image, thus generating a SRimage. As described above, since the phase of the enlarged image isfirst shifted so as to cancel the phase shift generated at the time ofgenerating the reduced image and the phase-shifted enlarged image isadded to the combined image, it is not only possible to remove morecertainly the errors included in the combined image but also to suppressgeneration of new errors, whereby the image quality of the SR image canbe improved.

Fourth Embodiment Configuration of Image Processor

When the LR image is in an interlaced format, an LR image of a top fieldand an LR image of a bottom field are alternately input to the imageprocessor. Thus, an LR image of one frame is obtained from two LR imageswhich are consecutive in time, that is, an LR image of a top field andan LR image of a bottom field which are consecutive.

Therefore, depending on a phase change pattern of the pixels on thecombined image which are sampled at the time of generating the reducedimage, there may occur a case where the phase of the pixels beingsampled is biased in the top field or the bottom field.

For example, when the phases of the pixels of the combined image whichare sampled every other field are shifted from the position of thereference pixel, the reference pixel will be sampled in any one of thetop field and the bottom field which appear every other field. In such acase, there is a concern that errors (noise components) are not detectedat a certain phase of the combined image but are accumulated.

Therefore, a phase change pattern for the top field and a phase changepattern for the bottom field may be prepared independently. In this way,it is possible to suppress accumulation of errors in the SR image of aparticular kind of field, namely the top field or the bottom field.

When the phases of the combined image at the time of generating thereduced image are changed independently for each kind of field, theimage processor may be configured as illustrated in FIG. 21, forexample. In FIG. 21, the same or corresponding portions as those in FIG.8 will be denoted by the same reference numerals, and descriptionthereof will be appropriately omitted.

The image processor 201 of FIG. 21 includes a switch 211, a phasecontroller 212, down-samplers 213-1 to 213-3, a phase controller 214,down-samplers 215-1 to 215-3, a phase controller 216, down-samplers217-1 to 217-3, a phase controller 218, and down-samplers 219-1 to219-3, in lieu of the phase controller 76, the down-samplers 77-1 to77-3, the phase controller 78, and the down-samplers 79-1 to 79-3 of theimage processor 61.

The switch 211 changes the output destination of the combined imagesupplied from the mixer 75 in accordance with the kind of the field(frame) of the LR image to be processed. That is to say, when thecurrent field being processed is a top field, the switch 211 suppliesthe supplied combined image to the phase controller 212. When thecurrent field is a bottom field, the switch 212 supplies the suppliedcombined image to the phase controller 216.

The phase controller 212 selects the phase of the combined image to beused for generating the reduced image and changes the output destinationof the combined image supplied from the switch 211 in accordance withthe selected phase. That is to say, the phase controller 212 suppliesthe supplied combined image to any one of the down-samplers 213-1 to213-3.

Each of the down-samplers 213-1 to 213-3 samples the pixels at apredetermined phase of the combined image supplied from the phasecontroller 212 and generates an image composed of such pixels, thusdown-sampling the combined image in the x-direction. Each of thedown-samplers 213-1 to 213-3 supplies the generated image to the phasecontroller 214. In the following description, when it is not necessaryto distinguish particularly between the down-samplers 213-1 to 213-3,they will be simply referred to as a down-sampler 213.

The phase controller 214 selects a phase of the combined image to beused for generating the reduced image and changes an output destinationof the image supplied from the down-sampler 213 in accordance with theselected phase. That is to say, the phase controller 214 supplies thesupplied image to any one of the down-samplers 215-1 to 215-3.

Each of the down-samplers 215-1 to 215-3 samples the pixels at apredetermined phase of the image supplied from the phase controller 214and generates a reduced image composed of such pixels. In this way, thecombined image is down-sampled in the y-direction.

Each of the down-samplers 215-1 to 215-3 supplies the generated reducedimage to the subtractor 80. In the following description, when it is notnecessary to distinguish particularly between the down-samplers 215-1 to215-3, they will be simply referred to as a down-sampler 215.

The phase controller 216 selects the phase of the combined image to beused for generating the reduced image and changes an output destinationof the image supplied from the switch 211 in accordance with theselected phase. That is to say, the phase controller 216 supplies thesupplied combined image to any one of the down-samplers 217-1 to 217-3.

Each of the down-samplers 217-1 to 217-3 samples the pixels at apredetermined phase of the combined image supplied from the phasecontroller 216 and generates an image composed of such pixels, thusdown-sampling the combined image in the x-direction. Each of thedown-samplers 217-1 to 217-3 supplies the generated image to the phasecontroller 218. In the following description, when it is not necessaryto distinguish particularly between the down-samplers 217-1 to 217-3,they will be simply referred to as a down-sampler 217.

The phase controller 218 selects a phase of the combined image to beused for generating the reduced image and changes an output destinationof the image supplied from the down-sampler 217 in accordance with theselected phase. That is to say, the phase controller 218 supplies thesupplied image to any one of the down-samplers 219-1 to 219-3.

Each of the down-samplers 219-1 to 219-3 samples the pixels at apredetermined phase of the image supplied from the phase controller 218and generates a reduced image composed of such pixels. In this way, thecombined image is down-sampled in the y-direction.

Each of the down-samplers 219-1 to 219-3 supplies the generated reducedimage to the subtractor 80. In the following description, when it is notnecessary to distinguish particularly between the down-samplers 219-1 to219-3, they will be simply referred to as a down-sampler 219.

The buffer 83 may be configured to store the SR image of the previoustwo fields so that an SR image which is two fields earlier than thecurrent field being processed is supplied to the motion vector detector72 and the motion compensator 73 by the buffer 83. In such a case, aprediction image is generated from the LR image of the top field and theSR image or from the LR image of the bottom field and the SR image.

However, the down-samplers 213, 215, 217, and 219 are configured todown-sample the combined image by generating an image composed of pixelsat a predetermined phase (position) of the combined image.

For example, as illustrated in FIG. 22, the down-samplers 213 and 215sample pixels at different phases based on a predetermined referencepixel. In FIG. 22, the rightward direction represents the x-direction,and the downward direction represents the y-direction. In addition, inthe figure, each circle represents each pixel on the combined image.

The down-sampler 213-1 samples a left adjacent pixel G82 of a referencepixel G81 which serves as a reference pixel on the combined image, asdepicted on the top left of the figure. The down-sampler 213-2 samplesthe reference pixel G81 on the combined image, as depicted on the topcenter of the figure. The down-sampler 213-3 samples a right adjacentpixel G83 of the reference pixel G81 on the combined image, as depictedon the top right of the figure.

The down-sampler 215-1 samples an upper adjacent pixel G92 of areference pixel G91 which serves as a reference pixel on the combinedimage which is down-sampled in the x-direction, as depicted on thebottom left of the figure.

The down-sampler 215-2 samples the reference pixel G91 on the combinedimage, as depicted on the bottom center of the figure. The down-sampler215-3 samples a lower adjacent pixel G93 of the reference pixel G91 onthe combined image, as depicted on the bottom right of the figure.

Moreover, the down-samplers 217-1 to 217-3 are configured to down-samplethe combined image by sampling the same pixels as the pixels which aresampled from the combined image by the down-samplers 213-1 to 213-3.

Similarly, the down-samplers 219-1 to 219-3 are configured todown-sample the combined image by sampling the same pixels as the pixelswhich are sampled from the combined image by the down-samplers 215-1 to215-3.

Furthermore, as illustrated in FIG. 23, for example, each of the phasecontrollers 212, 214, 216, and 218 selects the output destination of thecombined image so that the phases of the pixels to be sampled arechanged in a predetermined pattern.

In FIG. 23, the rightward direction represents time, and the numberabove each phase controller 212 or 216 represents a number thatidentifies the field (frame) of the combined image to be processed.Specifically, the frames are processed in the order from the 0th frameindicated by “0” to the 7th frame indicated by “7”. Moreover, thecharacters “T” or “B” to the right of the field number stands for thetop field and the bottom field, respectively, representing the kind ofthe field.

In addition, the bold line extending from the phase controller 212 tothe down-sampler 215 and the bold line extending from the phasecontroller 216 to the down-sampler 219 represent paths along which thecombined images are supplied.

In the example of FIG. 23, since the 0th field which is first processedis the top field, the combined image of the 0th field is supplied to thephase controller 212. Subsequently, the combined image is supplied fromthe phase controller 212 to the down-sampler 213-2, and the combinedimage supplied from the down-sampler 213-2 to the phase controller 214is then supplied to the down-sampler 215-2.

Since the 1st field subsequent to the 0th field is the bottom field, thecombined image of the 1st field is supplied from the switch 211 to thephase controller 216 Subsequently, the combined image of the 1st fieldis supplied from the phase controller 216 to the down-sampler 217-2, andthe combined image supplied from the down-sampler 217-2 to the phasecontroller 218 is then supplied to the down-sampler 219-2.

Similarly, since the 2nd field is the top field, the combined image ofthe 2nd field is supplied to the down-samplers 213-1 and 215-2.Moreover, since the 3rd field is the bottom field, the combined image ofthe 3rd field is supplied to the down-samplers 217-2 and 219-1.

Since the 4th field is the top field, the combined image of the 4thfield is supplied to the down-samplers 213-2 and 215-2. Since the 5thfield is the bottom field, the combined image of the 5th field issupplied to the down-samplers 217-2 and 219-2.

Since the 6th field is the top field, the combined image of the 6thfield is supplied to the down-samplers 213-2 and 215-3. Since the 7thfield is the bottom field, the combined image of the 7th field issupplied to the down-samplers 217-3 and 219-2.

In the 8th and later fields, the output destination of the combinedimage is changed in accordance with a predetermined pattern for eachkind of the field depending on whether a field being processed is thetop field or the bottom field.

As described above, in the top fields, a reduced image composed of thereference pixels of the combined image is generated every other field(frame). That is to say, the phases of the pixels sampled from thecombined image are shifted from the reference pixel every other field.Moreover, on the bottom fields, a reduced image composed of thereference pixels of the combined image is generated every other field(frame).

Operation of Image Processor

Next, with reference to the flowchart of FIG. 24, the image conversionprocess by the image processor 201 will be described. In FIG. 24, theprocesses of steps S131 to S134 are the same as the processes of stepsS11 to S14 in FIG. 11, and description thereof will be omitted.

At step S135, the switch 211 changes the output destination of thecombined image supplied from the mixer 75 depending on whether a fieldbeing processed is the top field or the bottom field. Specifically, whenthe field being processed is the top field, the switch 211 supplies thecombined image to the phase controller 212. When the field beingprocessed is the bottom field, the switch 211 supplies the combinedimage to the phase controller 216.

At step S136, the phase controller 212 or 216 selects the x-directionalphase of the combined image to be used for generating the reduced imagein accordance with a predetermined pattern, for example, the patterndescribed with reference to FIG. 23. The phase controller 212 or 216supplies the combined image supplied from the switch 211 to thedown-sampler 213 or 217 corresponding to the selected phase and causesx-directional down-sampling to be performed.

At step S137, the down-sampler 213 or 217 performs x-directionaldown-sampling on the combined image supplied from the phase controller212 or 216 and supplies an image obtained thus to the phase controller214 or 218.

At step S138, the phase controller 214 or 218 selects the y-directionalphase of the combined image to be used for generating the reduced imagein accordance with a predetermined pattern, for example, the patterndescribed with reference to FIG. 23. The phase controller 214 or 218supplies the combined image supplied from the down-sampler 213 or 217 tothe down-sampler 215 or 219 corresponding to the selected phase andcauses y-directional down-sampling to be performed.

At step S139, the down-sampler 215 or 219 performs y-directionaldown-sampling on the combined image supplied from the phase controller214 or 218 and supplies an image obtained thus to the subtractor 80. Inthis way, by down-sampling the combined image in the x and y-directions,a reduced image with the same resolution as the LR image is generated,and the generated reduced image is supplied to the subtractor 80.

Subsequently, the processes of steps S140 to S143 are performed, and theimage conversion process ends. Since the processes of steps S140 to S143are the same as the processes of steps S19 to S22 of FIG. 11,description thereof will be omitted.

In this way, the image processor 201 changes the output destination ofthe combined image in accordance with the kind of a field to beprocessed and changes the phases of the pixels to be used for generatingthe reduced image in accordance with a pattern prepared for each kind ofthe field.

As described above, the output destination of the combined image ischanged depending on the kind of the field, and the phases of the pixelsto be used for generating the reduced image are changed in accordancewith a pattern for each kind of the field. Therefore, it is possible toprevent the phases of the pixels used for generating the reduced imagefrom being biased to a certain phase in a certain kind of field.Accordingly, it is possible to suppress accumulation of errors in the SRimage of a certain kind of field and improve the image quality of the SRimage.

Although it has been described that the down-samplers 213, 215, 217, and219 down-sample the combined image by sampling certain pixels, thecombined image may be down-sampled by a filtering process.

Fifth Embodiment Configuration of Image Processor

Although it has been described that when the reduced image is generated,the combined image is first subjected to the x-directional down-samplingand then to the y-directional down-sampling, the x-directionaldown-sampling and the y-directional down-sampling may be performedsimultaneously.

In such a case, the image processor may be configured as illustrated inFIG. 25, for example. In FIG. 25, the same or corresponding portions asthose in FIG. 8 will be denoted by the same reference numerals, anddescription thereof will be appropriately omitted.

The image processor 251 of FIG. 25 includes a phase controller 261 andfiltering processors 262-1 to 262-5, in lieu of the phase controller 76,the down-samplers 77-1 to 77-3, the phase controller 78, and thedown-samplers 79-1 to 79-3 of the image processor 61.

The phase controller 261 selects the phase of the combined image to beused for generating the reduced image and changes an output destinationof the combined image supplied from the mixer 75 in accordance with theselected phase. That is to say, the phase controller 261 supplies thesupplied combined image to any one of the filtering processors 262-1 to262-5.

Each of the filtering processors 262-1 to 262-5 uses the combined imagesupplied from the phase controller 261 to calculate the pixels at apredetermined phase of the combined image supplied from the phasecontroller 261 by a 2-dimensional filtering process and generates areduced image composed of the calculated pixels. By this filteringprocess, the combined image is down-sampled in the x and y-directions.

Each of the filtering processors 262-1 to 262-5 supplies the generatedreduced image to the subtractor 80. In the following description, whenit is not necessary to distinguish particularly between the filteringprocessors 262-1 to 262-5, they will be simply referred to as afiltering processor 262.

The filtering process of the filtering processor 262 uses a plurality ofpixels on the combined image which are arranged in the x andy-directions as illustrated in FIG. 26, for example. The rightwarddirection in FIG. 26 represents the x-direction.

In the example of the image processor 121 described with reference toFIG. 13, it has been described that when the pixels at the phase of thereference pixel are calculated by the filtering process, the pixelswhich are arranged in the x-direction (or the y-direction) aremultiplied respectively with coefficients corresponding to the distancefrom the reference pixel, as indicated by an arrow Q61 in FIG. 26.

Moreover, in the example of the image processor 121, it has beendescribed that when pixels at a phase separated by a distance of a pixelor smaller from the reference pixel are calculated by the filteringprocess, the pixels which are arranged in the x-direction (or they-direction) are multiplied respectively with coefficients havingdifferent values for each pixel, as indicated by an arrow Q62.

In the example indicated by the arrows Q61 and Q62, each circlerepresents the coefficient multiplied with one pixel, and the verticaldirection represents the magnitude of the coefficients. That is to say,the coefficients indicated by the arrows Q61 and Q62 are the same as thecoefficients indicated by the arrow Q11 in FIG. 13 and the arrow Q33 inFIG. 15, respectively.

When the 1-dimensional filtering process using the coefficientsindicated by the arrows Q61 and Q62 is applied to a 2-dimensionalfiltering process, the pixels which are arranged in the x andy-directions are multiplied with coefficients as indicated by arrows Q63and Q64, whereby the pixel values of the pixels of the reduced image arecalculated.

In the example indicated by the arrows Q63 and Q64, each shadowed circlerepresents one coefficient multiplied with the pixel on the combinedimage, and circles representing such coefficients are arranged with thesame positional relationship as the pixels on the combined image.

In the example indicated by the arrows Q63 and Q64, the rightwarddirection and the downward direction in the left figure of the drawingrepresent the x-direction and the y-direction, respectively. Moreover,in the example indicated by the arrows Q63 and Q64, the rightwarddirection and the frontward direction in the right figure of the drawingrepresent the x-direction and the y-direction, respectively, and thevertical direction represents the magnitude of the coefficients.

The example indicated by the arrow Q63 represents the coefficients usedfor calculating the pixels at the phase of the reference pixel of thecombined image, and the three pixels by three pixels, which are arrangedin the x and y-directions, respectively, are multiplied with thecoefficients, whereby pixel values are calculated. In this example,among the 9 coefficients in total, the coefficient at the centerrepresents the coefficient multiplied with the reference pixel of thecombined image and has the largest value among the 9 coefficients.Moreover, the coefficients have values corresponding to the distance ofthe pixels being multiplied from the reference pixel, and thecoefficients multiplied with the pixels have the larger values as thepixels are located closer to the reference pixel.

The example indicated by the arrow Q64 represents the coefficients usedfor calculating the pixels at phases separated by a distance of a pixelor smaller from the reference pixel of the combined image, and the fourpixels by four pixels, which are arranged in the x and y-directions,respectively, are multiplied with the coefficients, whereby pixel valuesare calculated.

In this example, the central position of the 16 coefficients in total isthe phase of the pixels of a reduced image to be calculated, and thepixel multiplied with the coefficient which is located the closest tothe phase of the pixels of the reduced image is the reference pixel ofthe combined image. Moreover, the coefficients multiplied with thepixels of the combined image have the larger values as the pixels arelocated closer to the phase of the pixels of the reduced image which isto be calculated.

As described above, the filtering processor 262 multiplies a pluralityof pixels, which are located near the reference pixel and arranged inthe x and y-direction, with coefficients, whereby the pixel value ofeach pixel of the reduced image is calculated by a 2-dimensionalfiltering process. In this way, a reduced image composed of pixels atphases which are separated by a distance of a pixel or smaller from thereference pixel in the x and y-direction can be obtained more simply andquickly by one filtering process.

For example, the filtering processor 262-1 generates a reduced imagecomposed of pixels at phases which are shifted by ½ pixel in theopposite direction of the x-direction from the reference pixel of thecombined image. The filtering processor 262-2 generates a reduced imagecomposed of pixels at phases which are shifted by ½ pixel in thex-direction from the reference pixel of the combined image.

The filtering processor 262-3 generates a reduced image composed ofpixels at the phase of the reference pixel of the combined image. Thefiltering processor 262-4 generates a reduced image composed of pixelsat phases which are shifted by ½ pixel in the y-direction from thereference pixel of the combined image. The filtering processor 262-5generates a reduced image composed of pixels at phases which are shiftedby ½ pixel in the opposite direction of the y-direction from thereference pixel of the combined image.

Operation of Image Processor

Next, with reference to the flowchart of FIG. 27, the image conversionprocess by the image processor 251 will be described. In FIG. 27, theprocesses of steps S171 to S174 are the same as the processes of stepsS11 to S14 in FIG. 11, and description thereof will be omitted.

At step S175, the phase controller 261 selects the phase of the combinedimage to be used for generating a reduced image. For example, when thephase is changed in at least one of the x and y-directions in apredetermined pattern, the phase controller 261 selects a phase which isdetermined for the current frame being processed in accordance with thepattern.

The phase controller 261 supplies the combined image supplied from themixer 75 to the filtering processor 262 corresponding to the selectedphase and causes a reduced image to be generated.

At step S176, the filtering processor 262 performs a 2-dimensionalfiltering process on the combined image supplied from the phasecontroller 261 to generate a reduced image. Specifically, the filteringprocessor 262 multiplies the pixel values of several pixels around thereference pixel of the combined image supplied from the phase controller261 with coefficients stored in advance and calculates the sum of thepixel values multiplied with the coefficients, thus calculating thepixel value of each pixel of the reduced image.

When the reduced image is generated, the filtering processor 262supplies the generated reduced image to the subtractor 80. Subsequently,the processes of steps S177 to S180 are performed, and the imageconversion process ends. Since such processes are the same as theprocesses of steps S19 to S22 of FIG. 11, description thereof will beomitted.

In this way, the image processor 251 performs a super-resolution processon the input LR image and converts the LR image to the SR image. Whengenerating the SR image, the image processor 251 performs the2-dimensional filtering process on the combined image to generate thereduced image while changing the phases of the pixels of the combinedimage to be used for generating the reduced image from frame to frame.

As described above, since the combined image is generated by the2-dimensional filtering process, it is not only possible to shift thephases with an accuracy of a pixel or lower, but also to decreasefurther the amount of a phase shift in the subject between the reducedimage and the combined image. In this way, it is possible to decreasefurther the errors generated at the time of generating the SR image andimprove the image quality of the SR image. Moreover, it is possible toobtain the reduced image more simply and quickly by the 2-dimensionalfiltering process.

Furthermore, since the phases of the pixels of the combined image to beused for generating the reduced image are changed from frame to frame,it is possible to prevent accumulation of errors (noise components)generated at the time of generating the combined image, whereby theimage quality of the SR image can be improved.

Sixth Embodiment Configuration of Image Processor

When a reduced image having pixels corresponding to half the pixel countof the combined image is generated from the combined image, the phasesof the pixels of the combined image to be used for generating thereduced image are shifted for each frame by a distance of smaller than 2pixels, such as 1 pixel, ½ pixel, or ¼ pixel.

When a reduced image having pixels smaller than the half the pixel countof the combined image is generated from the combined image, since morepixels are thinned out, the phases of the pixels of the combined imageused for generating the reduced image may be shifted for each frame by adistance in units of pixels. For example, when a reduced image havingpixels corresponding to ⅛ of the pixel count of the combined image isgenerated, the phases may be shifted by a distance of smaller than 8pixels. Therefore, when the phases are shifted by a distance in units ofpixels such as 4 pixels, 2 pixels, or 1 pixel, the reduced image can begenerated by simpler processing.

Specifically, as illustrated in FIG. 28, when the phases of the pixelsto be used for generating the reduced image are shifted by apredetermined number of pixels in a predetermined direction from thereference pixel, the reduced image can be obtained more simply byshifting the phase (position) of each pixel of the combined image by thepredetermined number of pixels in the opposite direction of thepredetermined direction. In FIG. 28, each circle represents each pixelon the combined image.

For example, when the phases of the pixels of the combined image to beused for generating the reduced image are shifted by 2 pixels in theopposite direction of the x-direction, the phase of each pixel can beshifted by moving each pixel of the combined image in the x-direction by2 pixels as indicated by an arrow C11. In the example indicated by thearrow C11, in the figure, the upper line of plural circles representsthe combined image before the phases are shifted, and the lower line ofplural circles represents the combined image after the phases areshifted. In addition, in the figure, the rightward direction representsthe x-direction.

In the example indicated by the arrow C11, the phase of each pixel ofthe combined image is shifted by a distance of 2 pixels in thex-direction. Therefore, by generating an image composed of pixels of thephase-shifted combined image which are located at the same positions asthe reference pixel of the combined image before the phase shift, it ispossible to obtain an image composed of pixels which are located atpositions separated by 2 pixels in the opposite direction of thex-direction from the reference pixel of the combined image.

Similarly, when the phases of the pixels of the combined image to beused for generating the reduced image are shifted by 2 pixels in theopposite direction of the y-direction, the phase of each pixel can beshifted by moving each pixel of the combined image in the y-direction by2 pixels as indicated by an arrow C12. In the example indicated by thearrow C12, in the figure, the left line of plural circles represents thecombined image before the phases are shifted, and the right line ofplural circles represents the combined image after the phases areshifted. In addition, in the figure, the downward direction representsthe y-direction.

In the example indicated by the arrow C12, the phase of each pixel ofthe combined image is shifted by a distance of 2 pixels in the oppositedirection of the y-direction. Therefore, by generating an image composedof pixels of the phase-shifted combined image which are located at thesame positions as the reference pixel of the combined image before thephase shift, it is possible to obtain an image composed of pixels whichare located at positions separated by 2 pixels in the opposite directionof the y-direction from the reference pixel of the combined image.

In this way, when a reduced image composed of pixels at phases shiftedby a distance of a predetermined number of pixels from the referencepixel of the combined image is generated, the reduced image can beobtained more simply by shifting first the phase of each pixel of thecombined image in accordance with the phase and sampling the pixels atthe same positions as the original reference pixel.

In such a case, the image processor may be configured as illustrated inFIG. 29. In FIG. 29, the same or corresponding portions as those in FIG.8 will be denoted by the same reference numerals, and descriptionthereof will be appropriately omitted.

The image processor 291 of FIG. 29 includes a phase controller 301, adown-sampler 302, a phase controller 303, and a down-sampler 304, inlieu of the phase controller 76, the down-samplers 77-1 to 77-3, thephase controller 78, and the down-samplers 79-1 to 79-3 of the imageprocessor 61.

The phase controller 301 selects the phase of the combined image to beused for generating the reduced image and shifts appropriately the phaseof each pixel of the combined image supplied from the mixer 75 in thex-direction (more specifically, the direction parallel to thex-direction) in accordance with the selected phase. The phase controller301 supplies the phase-shifted combined image to the down-sampler 302 asnecessary.

The down-sampler 302 samples the pixels at a predetermined phase of thecombined image supplied from the phase controller 301, that is, thepixels at the same positions as the reference pixel of the originalcombined image, thus generating an image composed of such pixels. Inthis way, the combined image is down-sampled in the x-direction. Thedown-sampler 302 supplies the image (combined image) generated by thedown-sampling to the phase controller 303.

The phase controller 303 selects the phase of the image to be used forgenerating the reduced image and shifts appropriately the phase of eachpixel of the image supplied from the down-sampler 302 in the y-direction(more specifically, the direction parallel to the y-direction) inaccordance with the selected phase. The phase controller 303 suppliesthe phase-shifted image to the down-sampler 304 as necessary.

The down-sampler 304 samples the pixels at a predetermined phase of thecombined image supplied from the phase controller 303, that is, thepixels at the same positions as the reference pixel of the originalimage before the phase shift, thus generating a reduced image composedof such pixels. In this way, the combined image is down-sampled in they-direction. The down-sampler 304 supplies the reduced image generatedby the down-sampling to the subtractor 80.

Operation of Image Processor

Next, with reference to the flowchart of FIG. 30, the image conversionprocess by the image processor 291 will be described. In FIG. 30, theprocesses of steps S211 to S214 are the same as the processes of stepsS11 to S14 in FIG. 11, and description thereof will be omitted.

At step S215, the phase controller 301 selects the phase of the combinedimage to be used for generating the reduced image in accordance with apredetermined pattern and shifts appropriately the phase of each pixelof the combined image supplied from the mixer 75 in the x-direction inaccordance with the selected phase.

For example, when the selected phase is the same as the phase of thereference pixel, the phase controller 301 does not shift the phase ofeach pixel of the combined image but supplies the combined image to thedown-sampler 302 as it is. For example, when the selected phase isdifferent from the phase of the reference pixel, the phase controller301 shifts the phase of each pixel by moving each pixel of the combinedimage in accordance with the selected phase and supplies thephase-shifted combined image to the down-sampler 302.

At step S216, the down-sampler 302 samples the pixels at the sameposition as the reference pixel before the phase shift, of the combinedimage supplied from the phase controller 301, to generate an imagecomposed of such pixels, whereby the combined image is down-sampled inthe x-direction. The down-sampler 302 supplies the image obtained by thedown-sampling to the phase controller 303.

At step S217, the phase controller 303 selects the phase of the combinedimage to be used for generating the reduced image in accordance with apredetermined pattern and shifts appropriately the phase of each pixelof the combined image supplied from the down-sampler 302 in they-direction in accordance with the selected phase.

For example, when the selected phase is the same as the phase of thereference pixel, the phase controller 303 does not shift the phase ofeach pixel of the combined image but supplies the combined image to thedown-sampler 304 as it is. For example, when the selected phase isdifferent from the phase of the reference pixel, the phase controller303 shifts the phase of each pixel by moving each pixel of the combinedimage in accordance with the selected phase and supplies thephase-shifted combined image to the down-sampler 304.

At step S218, the down-sampler 304 samples the pixels at the sameposition as the reference pixel before the phase shift, of the combinedimage supplied from the phase controller 303, to generate an imagecomposed of such pixels, whereby the combined image is down-sampled inthe y-direction. The down-sampler 304 supplies the image obtained by thedown-sampling to the subtractor 80.

Subsequently, the processes of steps S219 to S222 are performed, and theimage conversion process ends. Since such processes are the same as theprocesses of steps S19 to S22 of FIG. 11, description thereof will beomitted.

In this way, the image processor 291 performs a super-resolution processon the input LR image and converts the LR image to the SR image. Whengenerating the SR image, the image processor 291 changes the phases ofthe pixels to be used for generating the reduced image from frame toframe and shifts the phase of each pixel of the combined image inaccordance with the phase, thus down-sampling the phase-shifted combinedimage and generating the reduced image.

As described above, by shifting the phase of each pixel of the combinedimage and then performing down-sampling to generate the reduced image,it is possible to obtain the reduced image more simply and quickly. Inthis case, since only one block is necessary for performing thedown-sampling in each direction of performing the down-sampling, it ispossible to decrease the size of the image processor 291.

Furthermore, since the phases of the pixels of the combined image to beused for generating the reduced image are changed from frame to frame,it is possible to prevent accumulation of errors (noise components)generated at the time of generating the combined image, whereby theimage quality of the SR image can be improved.

Although it has been described that the down-samplers 302 and 304generate the reduced image by sampling the pixels at a predeterminedposition of the image, the reduced image may be generated by a filteringprocess using the pixels at the predetermined position.

In such a case, the pixel values of the pixels at a predeterminedposition of the combined image having been subjected to the phase shiftby the phase controller 301 are multiplied with predeterminedcoefficients, and the value of the sum of the pixel values multipliedwith the coefficients is calculated. Then, the value of the calculatedsum is used as the pixel value of the pixel at the selected phase, andan image composed of the pixels at the selected phase of the combinedimage is generated. The image generated thus is used as the combinedimage which is down-sampled in the x-direction.

The above-described series of processing may be executed by hardware ormay be executed by software. When the series of processing is executedby the software, a program constituting the software is installed from aprogram recording medium in a computer integrated into an exclusivehardware or a general-personal computer which can execute variousfunctions by installing various programs in the computer.

FIG. 31 is a block diagram illustrating an exemplary hardwareconfiguration of a computer executing the above-described series ofprocessing by a program.

In the computer, a CPU (central processing unit) 501, a ROM (read onlymemory) 502, and a RAM (random access memory) 503 are connected to eachother via a bus 504.

The bus 504 is connected to an input/output interface 505. Theinput/output interface 505 is connected to an input unit 506 including akeyboard, a mouse, and a microphone; an output unit 507 including adisplay and a speaker, a recording unit 508 including a hard disk and anonvolatile memory, a communication unit 509 including a networkinterface, and a drive 510 driving a removable medium 511 such as amagnetic disc, an optical disc, an opto-magnetic disc, or asemiconductor memory.

In the computer configured thus, the CPU 501 loads the program recordedin the recording unit 508 into the RAM 503 via the input/outputinterface 505 and the bus 504 and executes the program, whereby theabove-described series of processing is executed.

The program executed by the computer (the CPU 501) may be provided bybeing recorded on the removable medium 511 which is a package mediumincluding a magnetic disc (including a flexible disk), an optical disc(e.g., CD-ROM (compact disc-read only memory) and DVD (digital versatiledisc)), an opto-magnetic disc, and a semiconductor memory.Alternatively, the program may be provided via wired or wirelesstransmission media such as local area network, the Internet, digitalsatellite broadcasting.

The program is installed in the recording unit 508 via the input/outputinterface 505 when the removable medium 511 is mounted on the drive 510.Furthermore, the program may be received by the communication unit 509via wired or wireless transmission media and installed in the recordingunit 508. In addition, the program may be installed in advance in theROM 502 or the recording unit 508.

The program executed by the computer may be a program executingprocessing in a time-sequential manner in accordance with the proceduresdescribed in this specification and may be a program executing theprocessing in a parallel manner or at necessary times such as inresponse to calls.

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Application JP 2009-012350 filedin the Japan Patent Office on Jan. 22, 2009, the entire content of whichis hereby incorporated by reference.

The embodiments of the present invention are not limited to theabove-described embodiments, but various modifications can be made in arange not departing from the gist of the present invention.

1. An image processor performing a super-resolution process ofconverting an input image with a first resolution to an output imagewith a second resolution higher than the first resolution with respectto a plurality of input images which are consecutive in time,comprising: prediction means for predicting the output image with thesecond resolution of a time being processed using the input image of thetime being processed and the output image obtained by performing thesuper-resolution process on an input image of a time earlier than thetime being processed; generation means for generating a reduced imagewith the first resolution composed of pixels at different phases of theprediction image, the phases being different from time to time, using aprediction image obtained by the prediction of the prediction means;difference calculation means for calculating a difference between theinput image of the time being processed and the reduced image; andaddition means for adding the difference which is up-sampled to thesecond resolution to the prediction image, thus generating the outputimage with the second resolution of the time being processed.
 2. Theimage processor according to claim 1, wherein the generation meanschanges a phase of each pixel of the prediction image to be used forgenerating the reduced image for each time in accordance with apredetermined pattern.
 3. The image processor according to claim 1,wherein the generation means includes: selection means for selecting aphase of each pixel of the prediction image; and sampling means forgenerating the reduced image by sampling pixels at the phase selected bythe selection means from the prediction image.
 4. The image processoraccording to claim 1, wherein the generation means includes: selectionmeans for selecting a phase of each pixel of the prediction image; andfiltering means for generating pixels at the selected phase by afiltering process using several pixels around a pixel of the predictionimage which is positioned at the phase selected by the selection means,thus generating the reduced image.
 5. The image processor according toclaim 1, wherein the generation means further comprising phase controlmeans for generating the reduced image composed of pixels which arepositioned at a phase separated by a predetermined distance in apredetermined direction from a predetermined reference phase of theprediction image, and shifting a phase of the difference up-sampled tothe second resolution by the predetermined distance in the predetermineddirection.
 6. The image processor according to claim 1, wherein: theinput image is an image of an interlaced format; and the generationmeans includes: switching means for changing an output destination ofthe prediction image depending on whether the input image of the timebeing processed is a top-field image or a bottom-field image; firstselection means for selecting a phase of each pixel of the predictionimage obtained from the input image of a top field which is output fromthe switching means; first sampling means for generating the reducedimage by sampling a pixel at the phase selected by the first selectionmeans from the prediction image; second selection means for selecting aphase of each pixel of the prediction image obtained from the inputimage of a bottom field which is output from the switching means; andsecond sampling means for generating the reduced image by samplingpixels at the phase selected by the second selection means from theprediction image.
 7. The image processor according to claim 6, whereinthe first and second selection means independently change the phase ofeach pixel of the prediction image used for generating the reduced imagefrom field to field in accordance with a predetermined pattern.
 8. Theimage processor according to claim 1, wherein: the generation meansincludes: phase control means for moving each pixel of the predictionimage by a predetermined distance in a predetermined direction to shifta phase of each pixel of the prediction image; and reduced imagegeneration means for generating the reduced image composed of pixels ata predetermined phase of the prediction image in which the phase isshifted by the phase control means, and the phase control means changesa direction of shifting the phase of each pixel of the prediction imagein each time in accordance with a predetermined pattern.
 9. An imageprocessing method for use in an image processor performing asuper-resolution process of converting an input image with a firstresolution to an output image with a second resolution higher than thefirst resolution with respect to a plurality of input images which areconsecutive in time, the image processor comprising: prediction meansfor predicting the output image with the second resolution of a timebeing processed using the input image of the time to be processed andthe output image obtained by performing the super-resolution process onan input image of a time earlier than the time being processed;generation means for generating a reduced image with the firstresolution composed of pixels at different phases of the predictionimage, the phases being different from time to time, using a predictionimage obtained by the prediction of the prediction means; differencecalculation means for calculating a difference between the input imageof the time being processed and the reduced image; and addition meansfor adding the difference which is up-sampled to the second resolutionto the prediction image, thus generating the output image with thesecond resolution of the time being processed, the method comprising thesteps of: causing the prediction means to generate the prediction image;causing the generation means to generate the reduced image composed ofpixels at a predetermined phase of the prediction image; causing thedifference calculation means to calculate the difference between theinput image and the reduced image; and causing the addition means to addthe difference to the prediction image to generate the output image ofthe time being processed.
 10. An image processing program for performinga super-resolution process of converting an input image with a firstresolution to an output image with a second resolution higher than thefirst resolution with respect to a plurality of input images which areconsecutive in time, the program causing a computer to executeprocessing comprising the steps of: predicting the output image with thesecond resolution of a time being processed using the input image of thetime being processed and the output image obtained by performing thesuper-resolution process on an input image of a time earlier than thetime being processed; generating a reduced image with the firstresolution composed of pixels at different phases of the predictionimage, the phases being different from time to time, using a predictionimage obtained by the prediction in the prediction step; calculating adifference between the input image of the time being processed and thereduced image; and adding the difference which is up-sampled to thesecond resolution to the prediction image, thus generating the outputimage with the second resolution of the time being processed.
 11. Animage processor performing a super-resolution process of converting aninput image with a first resolution to an output image with a secondresolution higher than the first resolution with respect to a pluralityof input images which are consecutive in time, comprising: a predictionunit configured to predict the output image with the second resolutionof a time being processed using the input image of the time beingprocessed and the output image obtained by performing thesuper-resolution process on an input image of a time earlier than thetime being processed; a generation unit configured to generate a reducedimage with the first resolution composed of pixels at different phasesof the prediction image, the phases being different from time to time,using a prediction image obtained by the prediction of the predictionunit; a difference calculation unit configured to calculate a differencebetween the input image of the time being processed and the reducedimage; and an addition unit configured to add the difference which isup-sampled to the second resolution to the prediction image, thusgenerating the output image with the second resolution of the time beingprocessed.